So I led a project recently to switch my team’s applications from Passenger to Unicorn. Unicorn is the latest hip web server for Ruby apps, with all kinds of high-profile people (most notably github) switching over to it.

Unicorn Basics

For those few who still don’t know, Unicorn is descended from Mongrel, and uses its Ragel HTTP parser. It’s a pure-ruby unix pre-fork server, which just means that it runs as a master + a stable of workers, and it forks and creates all the workers before requests ever come in (as opposed to on-demand forking). It’s essentially identical to Passenger in terms of performance, but has a couple of key advantages.

Unicorn vs. Passenger

Advantage #1: Invisible restarts. If you run Unicorn with “preload_app true”, then the master process loads all your application code, and creates copies to give to the workers. This makes spinning up workers basically instantaneous (because they don’t have to load the app themselves). This also means that when it’s time to deploy or otherwise restart, you just spin up a new master process. Once it’s done loading, it spins up its own workers and kills the old master. From the user perspective, the restart is instantaneous; their requests just start going to the new Unicorn instance. Compare this to Passenger, where each worker has to load its own copy, and does so in-line with the request. This is why the first couple of requests after a Passenger deploy suffer a huge performance hit, because your user has to wait for the whole application to load. Unicorn already has workers ready by the time it’s done restarting.

Advantage #2: It’s super unixy. Ryan Tomayko said it infinitely better than I ever could. This boils down to personal preference, because there’s no inherent technical advantage in being very unixy. But it is awesome, and one of the reasons why playing with Unicorn’s code is a lot of fun.

Advantage #3: Passenger is vaporware. Or, more accurately, the next version of Passenger is vaporware. The guys at Phusion do a lot of awesome work, but all the cool features they’ve been asked for have been pushed back to the next version. It’s been “in development” for a loooong time now, with absolutely no indications about when it will be released (one of my coworkers has asked them every four months or so, and received “eventually” as an answer every time). As an outsider, the impression I get is that they’re devoting all their time and energy to Union Station, and consequently not putting much effort into improving their other products.

What I Don’t Like About Unicorn

I love everything about what it does, but have a few gripes about what it doesn’t do. Not that it needs to be doing these things; they’re not necessarily its responsibility. But a) it doesn’t manage memory bloat and b) it doesn’t do any scaling.

Memory Bloat: Raise your hand if you’ve ever written a large web application that didn’t have any memory leaks. None whatsoever. Yeah, the guy in the back? He’s lying. Once your app hits sufficient size, you will get memory leaks, which will translate to bloated workers in Unicorn. Passenger doesn’t have this problem, because its auto-scaler winds up periodically killing workers and accidentally solving the bloat problem in the process.

Auto-Scaling: Unicorn’s built for dedicated server scenarios. You calculate the maximum load your hardware can handle, then pick your worker count. Unicorn will spin up that many workers and that’s what you get. Where this doesn’t work well is when you don’t strictly have dedicated servers. If, say, you have some backend scripts that run during low-load times, then it’s nice to be able to reclaim those resources.

How to Solve the Things I Don’t Like

Memory Bloat: I decided to opt for Weasel. It’s easy to use, simple to set up. My only complaint is that I couldn’t get its test suite to run under 1.9, at all. My fork runs the same code, just has a reworked test suite.

Auto-Scaling: I wrote Alicorn. It sends signals to the Unicorn master, telling it to add or remove workers as needed. It runs as a cron job, which makes it potentially less responsive than Passenger (but in practice about the same). It’s designed to be a little more conservative than Passenger, but it’s pretty tuneable. It comes with a built-in utility to help figure out what settings you need to be running with. Check out the code, and feel free to comment, (constructively) criticize, fork, pull-request, etc.

So I wrote a class awhile back that had to so some heavy-duty querying; interacting directly with a big-data system based on Postgres. No ORM in the way, so I didn’t have ActiveRecord to help me out (and the query was really hairy; I would’ve needed to write raw SQL anyway). So I found myself using pg, which is really a very pleasant and clean way to interact with Postgres DBs. You pass your query over and get back a PG::Result. PG::Result objects include Enumerable and implement a whole bunch of handy methods besides, but one thing they don’t have is a nice, convenient #size. The standard method for checking the size of your result set is #ntuples or #num_tuples, which (predictably) returns your number of tuples. The simplest way to get at the individual tuples is to pull them out as hashes, with 'column_name' => value pairs.

But if you’re writing unit tests around this, you might run into a problem. In my case, I was querying a specialized big-data system. No test version available. I suppose I could have booted Postgres, made a db, made a table, written factories against it, populated it with test data, and run my assertions that way. But that would have been a total pain, for what was quite a specialized little script. I also suppose I could have figured out how to directly initialize a PG::Result, but it’s not in the public API, and god knows what hurdles I would’ve run into in trying. Or, the simplest solution, I suppose I could’ve just used Mocha to mock out my PG::Result, and run my tests against that. But ew, gross. Have you ever tried to write a mock for an object that behaves like an array of hashes with a whole pile of extra methods? My code had some #each, some #size, some #map, and of course some #[] to view the values to run assertions against them. Picture that mock. Just imagine all the braces, all the brackets, how gross that’s going to look. (I’m not going to write it out; what are you, sick? Get out of here).

Of course, there’s a better way. Metaprogramming! The world’s best (worst) solution to all (very few) problems. But totally the right solution here. Because that awful, tangled, snarly mock object becomes pretty simple. Just build an array of hashes (i.e. our data; arrays of hashes are ugly, but that’s what the underlying object looks like here). Then define the specialized methods you’re going to need. Which, in my case, was just #num_tuples.

@mock_result = [{ "day" => nil, 
                  "client_ip" => "123.123.123.123",
                  "user_agent" => "SomeBannedUA"},
                { "day" => nil,
                  "client_ip" => "123.123.123.124",
                  "user_agent" => "googlebot"}, 
                { "day" => nil,
                  "client_ip" => "123.123.123.125",
                  "user_agent" => "TotallyAValidUA"}]
class << @mock_result
  def num_tuples
    self.size
  end
end

That easy! All of a sudden, we’ve got an object that includes all the behavior I need to run my code, with very little mocking required. Just stub PGConn.exec to return this baby, and voila! We can exercise our two hundred lines of code, with two hundred lines of tests, and aw yeah it’s basically painless. And because we defined the method on @mock_result’s eigenclass, it will vanish into the ether; no worries about having accidentally defined it on Array or something and risking some unexpected behavior further down in our tests. Isn’t Ruby awesome?

WARNING: WALL OF TEXT

In case you didn’t gather by the obnoxiously long list of testing tools there, this is about testing. Flash back to March-April 2012; I was working on upgrading four enormous Rails apps from REE to Ruby 1.9. I’d done most of the work, fixed most of the tests, checked a lot of corner cases. But one thing I still hadn’t quite done was get our CI server working properly.

Rafter has used TeamCity for its Continuous Integration server for a long time. The sf-ruby mailing list (on which I’m a dedicated lurker) occasionally discusses CI servers, and usually concludes that they all suck but TeamCity sucks the least. CI servers are a fantastic idea in theory, and everyone loves having one; but they’re always a pain to set up, the configuration is always miserable, they’re always finicky, and they’re always easy to break. I’ve never personally used any but TC, so I don’t actually have strong feelings about whether or not it’s better than the competition. But it’s what we use, and our one attempt to switch away (to Jenkins) ended badly, so TC it is.

Our TC server ran builds for unit, functional, and integration tests for all four of our rails apps, and also ran the test suite for a gem (Hereafter “The Gem”) we maintain. The Gem has some pretty critical logic; bugs in it will break large, important pieces of our system, and cause Very Bad Things™ to happen. TeamCity’s test runners are weird; they’re deliberately obfuscated, injected at runtime, and completely impossible to debug. And they interact with test suites in funny ways.

Right, there’s your backstory. Here’s the problem. We were running everything on REE, with test-unit + shoulda for our framework. We had ci_reporter and JetBrains’ own Ant JUnit Reporter to generate, read, and parse test results. Everything worked swimmingly when you ran it on your local machine. The following paragraphs chronicle various attempts to get update this lovely system to Ruby 1.9.

Switched to 1.9 – Used test-unit + ci_reporter + Ant Junit Reporter

On TeamCity, this gave us functioning Rails test suites with machine-parsed results; but wouldn’t run tests for The Gem. Locally, everything worked fine.

Upgraded test-unit

Upgraded test-unit from 2.1.1 to 2.4.8 to try and fix tests for The Gem, on TeamCity. Everything was now wonderful on TeamCity! Huzzah! BUT. When running tests on a local machine, everything worked swimmingly, with no output. That nice little row of dots that test-unit used to print across the screen, giving you a rough gauge of progress? Gone. Now you just stared at a blinking cursor until the suite was done, and spat everything out at once. Annoying. As. Fuck.

Removed ci_reporter

Didn’t change a damn thing. (Later, I realized that this was always meaningless in our setup, as TeamCity was relying on the Ant JUnit reports. I’m not even sure why it was in there in the first place.)

Switched from test-unit to minitest

minitest is the new hotness in Ruby 1.9; it’s smaller, sleeker, faster, and sexier than test-unit. Less full-featured, though, lacking support for pluggable test reporters and with a stripped-down library of assertions. This caused a fair bit of headache, as I had to clean up some assertions that went away, and add in replacements for assertions that changed (because, infuriatingly, minitest implements some key assertions backwards with respect to the test-unit versions). But it is faster, and significantly cleaner, so I deemed it worth the effort to switch. This got everything working locally, with dots (woohoo!). But now it broke everything on TeamCity. TeamCity depended on test-unit’s pluggable-reporter support, and so (while it reported red/green successfully, as that just depended on the success or failure of the rake task) it didn’t parse the results at all. For a suite totaling about 14,000 tests running over 60,000 assertions, that kind of sucked.

Added minitest-reporters

Discovered and added minitest-reporters, a sweet gem that is mostly a dirty monkeypatch for minitest. It aims to give minitest the pluggable reporter support it lacks. Sounds perfect, right? It mostly is. This got all our tests running well locally (with sweet, swappable reporters, including a dynamically-updating progress bar with as-it-happens failure logging). And it got our tests running on TeamCity! AWESOME. But wait! What’s this? It took FOREVER. The TeamCity builds took anywhere between four and ten times as long as they used to. Just ludicrously, ridiculously slow. Unusable.

Added minitest-ci (on TeamCity only)

minitest-ci is another attempt to solve these problems. It’s another monkeypatched tool, that hooks into minitest and writes out JUnit reports, which (essentially) all CI systems know how to read. It actually worked, mostly, and in reasonable time. But the machine-parsed reports were pretty substandard; the test counts were extremely low, and it was apparent that something was not running as intended.

Rolled my own

Right, so that was kind of an odyssey. An enormously large amount of tools, nothing works quite right, but it’s all easier than doing it myself. Yeah? No. I opted to write my own custom JUnit reporter for minitest-reporters. Wasn’t too hard, wasn’t too long (just over 100 lines), and I had the benefit of minitest-ci and ci_reporter to learn from. I realized that minitest-ci’s biggest problem came from poor xml escaping; that led me to use the excellent, high-quality, always-reliable builder for xml. And I found some code in ci_reporter around handling test files with duplicate names; my approach wasn’t nearly as robust as Nick Sieger’s, but worked in a pinch and involved fewer moving parts. Now my team has beautiful local tests, beautiful TeamCity tests, and accurate test counts everywhere. And Alex Kern was totally cool about merging that into minitest-reporters (though I think he mixed me up with the guy who’d filed an issue asking for the feature :p), so now everyone can have beautiful TeamCity tests.

Good God that took a long time

Yes. Yes it did. And I really felt like a crazy man toward the end, given that I was doing the whole thing FOR THE DOTS. Literally. The first solution I tried had everything except aesthetically-pleasing dots. But it was worth it! It was fun, and interesting, and led me to discover, learn about, and then participate in some cool projects. Nothing quite like a week or two of diving very, very deeply into a problem, learning far more than you ever wanted to, and coming out with a solution you’re happy with.

FOR THE DOTS.

Just a quick post, mostly filler to say that I do intend to actually use this thing. It has sat here for over two years now, mostly gathering dust, with perennial “I-will-use-it!” oaths being sworn. And who knows, maybe that’s what this is too. But damn it all if I’m not going to do it in (obscure, readerless) public. My post ideas are being kept on the github issues that goes along with the blog. I have a friend who’s promised to nag me to make me post. The spambots have finally (fingers crossed) given up plaguing the comments in that first post I wrote. And I actually do have an interesting, potentially useful piece of OSS I plan on releasing to the world pretty soon (well, technically the code’s already public, poke around on my github account if you want to find it; but I’ll be, you know, talking about it).

So yes, blogging. This is a thing. I will do it! Just see if I don’t.

So I recently came across Zach Holman’s blog; he’s a very clever coder working at Github, and his screencast on Automating Your Inefficiencies has been making the rounds. Picking through his blog archives, though, I found this, which led me to the wonderful world of versioned dotfiles. To quote Zach:

Think about it: these are the files, aliases, and executables that programmers make for themselves to make their lives more productive.

Versioning them gives you several advantages.

• You’ve got version control! You can be fearless when making changes, because it’s easy to recover.
• You’ve got branching: you can play with variations, persist alternate versions, keep different files for different contexts.
• They’re conveniently packaged. It’s easy to redistribute them, easy to share, easy to configure an environment on a new box.

And making them public gives you more.

• New tricks! If someone forks you and then does some cool stuff, you can find it and use it yourself.
• Reputation! Everything you give away is something other people can use. And every time somebody uses something you wrote, there’s one more person who knows who you are, why you’re the shit, and why they want to work with you in the future.
• Goodness! There are plenty of awesome selfish reasons to give things away. There are unselfish ones too. And protip: they’re also awesome.

So grab my dotfiles, fork ‘em, play with ’em, let me know what’s broken and what’s awesome and what sucks. The readme on the repo will tell you how to use them and what’s unique about them, but in a nutshell: I’m keeping separate branches for OS X and Ubuntu, aiming for a sexy bash prompt, convenient git settings, and the world’s best version of the One True Editor™.

I realize now that blogging is the sort of business that doesn’t necessarily come naturally. For one, it’s just a big chunk of work, that I don’t necessarily immediately want to do. It might make things easier if I made the font larger on the admin-interface for the site, or if I had a decent head of steam behind my posts. But on the theory that discipline will do the trick, I’m going to try and force myself to actually use this, given that I’ve gone to all the trouble to get it working. Incidentally, family/friends, if anyone else wants to jump on and start posting here as well, feel free!

I was struck earlier today by a thought about the “code as craft” mentality that many Rubyists strive for. This came out of a conversation from the office a couple of weeks ago, when several of us were discussing our attitudes towards coding and refactoring, and the always-crucial balance between shipping a product and writing beautiful code. So I’m going to put a couple of (extreeeeemely generalized) postulates out here:

#1: Rubyists tend to pay more attention to aesthetics in their code. I don’t mean the way the product looks and feels, but the way the code itself looks and feels; there is an inherent value in producing elegantly-written software.

#2: Rubyists value those aesthetics; our brains read “elegantly-written software” as “well-written software”. Beauty implies cleanliness, which means readability and maintainability. So there’s a self-reinforcing pattern; not only do we feel dissatisfied producing ugly code because it’s ugly, but we suspect that ugly code means a technical debt we’re going to have to repay later, with interest.

#3: Non-Rubyists are more likely to sacrifice aesthetics in favor of ease-of-work. This sounds dismissive, but it’s not meant to be; I know I personally have a tendency to get so hung up on keeping the structure pretty that it keeps me from embracing the fastest and simplest way to get the job done. When you need to ship, sometimes you don’t have the option of elegance.

Those are all pretty standard thoughts, really. But what occurred to me earlier today is that the language itself may be the reason we value aesthetics more than programmers working with other language. I realized that it’s not really that I value aesthetics more than a C programmer would; but when I write Ruby, I value aesthetics more than when I write C. And when I’m working in Python, it’s somewhere in the middle. It’s something coming out of the beauty and the expressiveness of the language itself. I suspect that more Rubyists care about beauty because, frankly, it’s easier for their code to come out beautiful. I realized that Ruby’s syntax has wormed its way into my brainstem; the cleanness of the language reshapes my perceptions of all programming.

Caveat: I am totally not claiming that Ruby is the best language, or even the prettiest. While it’s my favorite, it’s only one tool in any proper toolbox. If anything, the aesthetic-coding mentality is a trait of great programmers and great language experts; because Ruby’s an easy language to get fluent with, and to understand, maybe it’s bringing that mentality down to a more accessible level of skill.

So I was going to write a first post full of long-winded invective about the BSA’s annual software piracy report. For those who don’t know, the Business Software Alliance is an industry trade group, whose members include Microsoft, Apple, Cisco, Intel, and a whole pack of other companies you’ve probably heard of. The BSA’s primary mission is convincing employees to rat out their bosses, when companies use unlicensed software. They’ve just released their 2009 Piracy Study, and it’s a pack of lies as usual. I won’t bother breaking it down, because it’s an exercise in futility; they don’t even bother to pretend like they have sources, or that their methodology for estimating piracy rates is sound, they just whip up sound bites for the press.

What’s a whole lot more exciting than the BSA is the Humble Indie Bundle. Wolfire Games is an indie game studio that decided to try out a different kind of sale. They partnered with four other game studios to offer five indie games in a pay-what-you-want bundle. That is, customers are required to pay at least $0.01, and get to download all five games, DRM-free, as many times as they want. On top of that, the game developers aren’t the only beneficiaries; a slice of the sale’s profits is going straight to the Electronic Frontier Foundation, and another slice is going straight to Child’s Play, the gamer charity set up and run by Penny Arcade. Awesome, no? Earlier today, the bundle broke one million dollars (and are now approaching 1.1 mil), and in celebration the sale’s been extended by another week and four of the developers announced they’ll be releasing the source code behind their games.

Awesome, no? This is a clear-cut case of developers thinking outside of the traditional “us vs. them” mindset that the big publishing companies seem to be stuck in. And what’s even cooler is the fact that the guys at Wolfire have been blogging about the sale all along, releasing a lot of really interesting information. For instance, by rough estimates, they’re guessing that despite the sale being pay-what-you-want, they’re still getting a 20-25% piracy rate. And in the early days of the sale, Linux users were choosing to pay more than twice what Windows users were giving (it’s evened out a little more since, but is still pretty lopsided). And despite Linux and OS X representing very small percentages of the world’s PC markets, they’re both representing a huge share of the bundle’s revenue. Lots of interesting conclusions to be teased out here.

Also exciting is the source code release. As an open-source developer myself, you can expect that I’m a huge fan! And, as with so many successful open-source projects, they’re releasing it under the GPL, with similar terms governing reuse of the game’s art assets. Obviously, the people who will enjoy this the most are each game’s die-hard dedicated fans (who are a little less likely to be contributing to the Humble Bundle, incidentally, since they’ve already bought some of these games previously). The folks who’ve been modding these games for years are going to be kids in a candy shop.

Compare to the BSA, making up ludicrous statistics for how much the software industry’s “losing to piracy” and equating every pirated copy with a lost sale (which is the kind of laughable mistake one would flunk a first-year statistics course for). Obviously, the game publishing giants wouldn’t get anywhere near the traction with a pay-what-you-want sale as indie developers can, because nobody can muster up as much sympathy for a giant faceless corporation as for a few guys doing what they love. But it’s still demonstrative. When are the major software companies (and here I’m including Microsoft and Adobe, as well as EA and Activision) going to realize that they can get a lot farther on the goodwill of their customers than they can by making them into criminals?