When you are running a distributed service in a datacenter, you encounter a lot of interesting problems. At Audiogalaxy, I ran into all the standard application level bugs, crashes, and race conditions. Once we had a certain number of machines, we even had to deal with flaky memory, disks, and networking cards. But all of that was pretty typical compared to the weirdest bug I ever had to deal with – the one that was caused by Quake III Arena.
Tag Archive for 'Audiogalaxy'
This is the final article in my three part series about building Audiogalaxy. If you haven’t already, you may want to read part one and part two first.
2002
By 2002, things were going along quite nicely from a technology standpoint. Running the C backend on the hardware we had purchased for the Java version meant we were very much below our capacity, and we made it past 1 million simultaneous clients. I was pleased that I had learned the right lessons from V1 and designed a solid architecture.
I was also feeling good about the other major service I looked after — search. Audiogalaxy had sponsored my senior engineering project in 2000 — a high performance MySQL indexing service. That, combined with a caching service I built in 2001, had solved the search problem. We were handling 80 million page views every day, most of them searches.
This is the second article in my three part history about building Audiogalaxy.com. You should probably read the first one first.
2000
I came back from my Christmas break feeling less burnt out. I focused on designing a backend that could handle 100,000 simultaneous Satellites and then started building it. To free me up from working on the client, Michael bought a copy of the Steven’s networking book and started working on a C version of the Satellite core. And David hired Kennon Ballou to help build the next-generation web interface.
The new backend and client went live in April, using my humble website from V1. Traffic started growing steadily, and by the end of May, we had about 3,000 clients connected at peak times. Sometime around the end of July, there was a Napster injunction scare, which pushed us over the 8,000 mark. We released version 0.6 of the client and David’s beautiful new website in September. At that point, our peak load started increasing by thousands of users every week.
Reading all the SXSW press always makes me nostalgic for my time in Austin and the 3 years I spent working at Audiogalaxy.com, a music startup. Here is a history of the company from the perspective of the first full-time programmer. This is the first of three parts.
Getting the Job
My career at Audiogalaxy got started by chance one spring afternoon in 1999. At the time, I was a computer-engineering student with 4 more classes to finish before I graduated that fall. Like most of my friends, I was planning on taking an entry-level engineering job at one of the big companies in Austin, but things didn’t end up turning out that way.
I ran into Michael Merhej on the steps of the Electrical and Computer Engineering building and stopped to chat about the business I had heard he had started. Michael and I had been lab partners in an electrical engineering class a few semesters before, and he had struck me as a high energy guy with an amazing capacity to focus on the things he was interested in. We talked a little bit about Audiogalaxy, and he told me that they really needed some web developers. A music website sounded a lot more fun than working for the engineering career center like I had done the previous year, so I told him I was interested.
Tackling a big, new programming challenge frequently means that you don’t actually understand the problem until after you solve it. Because of that, one of the most important things to do after nailing down a solution is to think about better solutions. Post-mortems seem to be more popular from a “how could we have managed this project better” angle, but the technical aspect should not be overlooked.
Sure, constant criticism of everything you do may not make you a happier person, but even if you don’t actually implement your new ideas, reevaluating your designs will make you a stronger architect. Just try to keep the habit from spilling over too much into your personal life. 
Continue reading ‘Lessons Learned Scaling the Audiogalaxy Search Engine’
As I mentioned before, search was one of most interesting problems I worked on at Audiogalaxy. It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. It was frequently hard to design for more than 10 or 100x our current traffic (and our growth was so dramatic that there wasn’t really ever time to spend more than a few weeks on the problem), so it wasn’t until the 4th major iteration on the cluster design that I really considered the scale problems to be solved.
Continue reading ‘Design details of Audiogalaxy.com’s high performance MySQL search engine’
When I first started working at Audiogalaxy, the site was running on a handful of cobbled together machines and serving about 100K page views per day. By the time we shut it down in 2002, we had scaled it to 80M daily page views and around 35M user accounts. Those were big numbers back then, and the 3 years I spent working there were a lot of fun.
Continue reading ‘Scaling Audiogalaxy to 80 million daily page views’
I went to a meeting run by the Seattle Tech Startup folks a few weeks ago. Even though I’m not thinking about doing another startup right now, I was glad to see the enthusiasm of all the other people who are. Because I love seeing the new ideas that come out of startups, I really hate seeing them fail as a result of them making the same silly mistakes. So, the collaboration that the STS meetings and the associated mailing list promote really put a smile on my face.
I’ve played a big part in two successful startups, and my two had very different flavors. Audiogalaxy was a rocket-ship ride that didn’t let up for three years until it all came crashing down due to a lawsuit. We had traffic from the minute we turned on the Satellite, and all we had to do was scale it up as quickly as we possibly could. FolderShare, on the other hand, sometimes felt like a continuous series of failures until we had massive success all at once–a glowing review from Walt Mossberg while we’re negotiating our acquisition by Microsoft! The wow starts now, all right.
Now that I’ve got some free time, one of the goals for this blog is to reflect a little bit on my experiences and what I might change next time. Stay tuned.
