When you are running a distributed service in a datacenter, you encounter a lot of interesting problems. At Audiogalaxy, I ran into all the standard application level bugs, crashes, and race conditions. Once we had a certain number of machines, we even had to deal with flaky memory, disks, and networking cards. But all of that was pretty typical compared to the weirdest bug I ever had to deal with – the one that was caused by Quake III Arena.
Archive for March, 2008
Next up in the toolbox series is an idea so good it deserves an entire article all to itself: consistent hashing.
Let’s say you’re a hot startup and your database is starting to slow down. You decide to cache some results so that you can render web pages more quickly. If you want your cache to use multiple servers (scale horizontally, in the biz), you’ll need some way of picking the right server for a particular key. If you only have 5 to 10 minutes allocated for this problem on your development schedule, you’ll end up using what is known as the naïve solution: put your N server IPs in an array and pick one using key % N.
I kid, I kid — I know you don’t have a development schedule. That’s OK. You’re a startup.
Continue reading ‘Programmer’s Toolbox Part 3: Consistent Hashing’
This is the final article in my three part series about building Audiogalaxy. If you haven’t already, you may want to read part one and part two first.
2002
By 2002, things were going along quite nicely from a technology standpoint. Running the C backend on the hardware we had purchased for the Java version meant we were very much below our capacity, and we made it past 1 million simultaneous clients. I was pleased that I had learned the right lessons from V1 and designed a solid architecture.
I was also feeling good about the other major service I looked after — search. Audiogalaxy had sponsored my senior engineering project in 2000 — a high performance MySQL indexing service. That, combined with a caching service I built in 2001, had solved the search problem. We were handling 80 million page views every day, most of them searches.
This is the second article in my three part history about building Audiogalaxy.com. You should probably read the first one first.
2000
I came back from my Christmas break feeling less burnt out. I focused on designing a backend that could handle 100,000 simultaneous Satellites and then started building it. To free me up from working on the client, Michael bought a copy of the Steven’s networking book and started working on a C version of the Satellite core. And David hired Kennon Ballou to help build the next-generation web interface.
The new backend and client went live in April, using my humble website from V1. Traffic started growing steadily, and by the end of May, we had about 3,000 clients connected at peak times. Sometime around the end of July, there was a Napster injunction scare, which pushed us over the 8,000 mark. We released version 0.6 of the client and David’s beautiful new website in September. At that point, our peak load started increasing by thousands of users every week.
Reading all the SXSW press always makes me nostalgic for my time in Austin and the 3 years I spent working at Audiogalaxy.com, a music startup. Here is a history of the company from the perspective of the first full-time programmer. This is the first of three parts.
Getting the Job
My career at Audiogalaxy got started by chance one spring afternoon in 1999. At the time, I was a computer-engineering student with 4 more classes to finish before I graduated that fall. Like most of my friends, I was planning on taking an entry-level engineering job at one of the big companies in Austin, but things didn’t end up turning out that way.
I ran into Michael Merhej on the steps of the Electrical and Computer Engineering building and stopped to chat about the business I had heard he had started. Michael and I had been lab partners in an electrical engineering class a few semesters before, and he had struck me as a high energy guy with an amazing capacity to focus on the things he was interested in. We talked a little bit about Audiogalaxy, and he told me that they really needed some web developers. A music website sounded a lot more fun than working for the engineering career center like I had done the previous year, so I told him I was interested.
Tackling a big, new programming challenge frequently means that you don’t actually understand the problem until after you solve it. Because of that, one of the most important things to do after nailing down a solution is to think about better solutions. Post-mortems seem to be more popular from a “how could we have managed this project better” angle, but the technical aspect should not be overlooked.
Sure, constant criticism of everything you do may not make you a happier person, but even if you don’t actually implement your new ideas, reevaluating your designs will make you a stronger architect. Just try to keep the habit from spilling over too much into your personal life. 
Continue reading ‘Lessons Learned Scaling the Audiogalaxy Search Engine’
