As I mentioned before, search was one of most interesting problems I worked on at Audiogalaxy. It was one of the core functions of the site, and somewhere between 50 to 70 million searches were performed every day. At peak times, the search engine needed to handle 1500-2000 searches every second against a MySQL database with about 200 million rows. It was frequently hard to design for more than 10 or 100x our current traffic (and our growth was so dramatic that there wasn’t really ever time to spend more than a few weeks on the problem), so it wasn’t until the 4th major iteration on the cluster design that I really considered the scale problems to be solved.
Continue reading ‘Design details of Audiogalaxy.com’s high performance MySQL search engine’
When I first started working at Audiogalaxy, the site was running on a handful of cobbled together machines and serving about 100K page views per day. By the time we shut it down in 2002, we had scaled it to 80M daily page views and around 35M user accounts. Those were big numbers back then, and the 3 years I spent working there were a lot of fun.
Continue reading ‘Scaling Audiogalaxy to 80 million daily page views’
Following up on my first post, What should be in your programming toolbox, here are a few more ideas from my list:
Unrolled Linked Lists
Wikipedia has more details, but essentially, an unrolled linked list is great because it will give you better performance than a regular linked list, is more cache friendly, and will probably have much less overhead. The basic idea is to store an array of elements at each node rather than a single one. This keeps your pointers closer together, which will make your cache happy when you are iterating through your items.
Continue reading ‘5 more essentials for your programming toolbox’
I really enjoy writing the code that makes systems like Audiogalaxy and FolderShare run. Getting into the zone and really getting some good work done is a great experience, but remains my second favorite aspect of the job. For me, the best part is the design phase before the real coding starts. At that point, everything is totally fluid and malleable. I’m making the decisions that I’m going to live with for the next few years, and putting some extra cleverness or flexibility into the system can have huge payoffs.
Something I’ve found very helpful at this phase is a “programming toolbox” — a simple list of good ideas and approaches to different problems. When I’m stuck on a problem, or trying to generate a new approach to something, it can be helpful to flip through the list. Most of the ideas won’t apply, but sometimes it will spark something novel. To keep this list from getting too unwieldy, I’ll post a few at a time as I write them up.
Continue reading ‘What should be in your programming toolbox?’
I went to a meeting run by the Seattle Tech Startup folks a few weeks ago. Even though I’m not thinking about doing another startup right now, I was glad to see the enthusiasm of all the other people who are. Because I love seeing the new ideas that come out of startups, I really hate seeing them fail as a result of them making the same silly mistakes. So, the collaboration that the STS meetings and the associated mailing list promote really put a smile on my face.
I’ve played a big part in two successful startups, and my two had very different flavors. Audiogalaxy was a rocket-ship ride that didn’t let up for three years until it all came crashing down due to a lawsuit. We had traffic from the minute we turned on the Satellite, and all we had to do was scale it up as quickly as we possibly could. FolderShare, on the other hand, sometimes felt like a continuous series of failures until we had massive success all at once–a glowing review from Walt Mossberg while we’re negotiating our acquisition by Microsoft! The wow starts now, all right.
Now that I’ve got some free time, one of the goals for this blog is to reflect a little bit on my experiences and what I might change next time. Stay tuned.
