Spiteful.com

When I first started working at Audiogalaxy, the site was running on a handful of cobbled together machines and serving about 100K page views per day. By the time we shut it down in 2002, we had scaled it to 80M daily page views and around 35M user accounts. Those were big numbers back then, and the 3 years I spent working there were a lot of fun.

Alexa may be a pack of lies sometimes, but it has to mean something that it claims we were in the top 20 or 30 websites on the web:

When we were still on the positive side of that curve, we had tons of interesting problems (not surprisingly, dealing with exponentially shrinking traffic doesn’t take quite the same level of dedication as exponentially increasing traffic). At the peak, our hardware distribution looked something like this:

• Web Servers: ~45 dual proc PIII’s with 2GB of RAM
• Databases: ~40 dual proc PIII’s with 1-2GB of RAM
• Search: ~8 misc machines with fast hard drives
• Satellite Servers: 300+ single proc PIII’s with 1GB RAM
• Misc: About a dozen other machines with assorted specs for message passing, mail, etc.

Scaling the Databases
We scaled our databases out using a number of strategies. For our most heavily accessed data set, we had an extremely good read/write ratio, so we were able to fan out to about 20 slaves from a single master. This particular database had several hundred million rows, which challenged the limits of our hardware (periodically, we had to clean out stale data when it got too large), so one trick we used was index-segmentation. Different sets of slaves had different indexes, and our database access layer could pick a different cluster based on the necessary index. Specifically, the tables in this database generally had an ID and a string, but the index on the string was only necessary for some queries. So, on some slaves we simply didn’t have the string index. This allowed those machines to keep the entire ID index in memory, which was a huge performance boost.

We used sharding to scale our databases in other areas. For example, each user account had a column that indicated which database that user’s requests for songs were stored in. This prevented us from doing some aggregate queries easily, but it allowed us to easily scale out our capacity to hold user requests.

Sessions
As any site scales beyond a single web server, tracking user sessions is a standard problem. We used a set of MySQL servers with in-memory databases for our sessions. We partitioned across them by simply taking a few bits from the session ID and using that to index into our machine list. If I was going to build this again, I would seriously look into using MySQL Cluster for the following reasons:

• It would be a compartmentalized, easy to monitor area to evaluate how the technology works and gain experience dealing with problems
• It would simplify the logic behind building stats about who is currently browsing the website (the script that did that would not have to be partitioning aware)
• We could restart databases without losing sessions and forcing users to login again. Knowing we won’t lose sessions may allow us to keep more interesting information in them.

Search
I’m going to cover search in more detail in another article, but our search engine had the following requirements:

• Search ~200M songs in a MySQL database
• Adjust the order of results based on real time file availability
• Handle 2000 searches per second at peak times.

Our actual search algorithm was more complicated than our schema, so even if it had existed at the time, the MySQL full text index wouldn’t have been a turnkey solution. Search was one of the most interesting areas of work for me, and it took several iterations before I completely solved the scalability problems. Not surprisingly, it ended up being a completely custom solution.

Web Servers
Our entire web site was built on PHP, and as I mentioned above, we had about 45 web servers. I believe we used the RedHat High Availability solution as a software load balancer. To keep CPU and script execution time down to a reasonable level, we used a very new and flaky PHP accelerator to cache compiled versions of the PHP scripts. Finally, we found that gzip compression was a big win — we used over 100 megabits of bandwidth all of the time, so every little percent we could cut out saved us money.

One neat management hack we used was a webpage with 45 frames that loaded the front page of the website off each server. Refreshing the page after doing any sort of maintenance made it easy to spot broken servers.

Satellite Servers
We used about 300 machines to handle all of the incoming connections from users running the Satellite, which made it the largest and most expensive piece of our infrastructure. I’ll go through this in more detail in a later article as well, but this was a primarily single-threaded server written in C using /dev/epoll as our IO model. As a whole, the cluster handled about 1.1M simultaneous connections at peak times. Individual servers could handle 6-8K connections each, which left us with plenty of spare capacity.

One of the less technical challenges I did have to deal with once we started turning the system off was unracking the servers and bringing them back to the office. If you imagine 2 guys who each own a car limited to holding about 15 servers, I’ll let you do the math to figure out how many trips it took to construct this:

I’ve got more details about how we built Audiogalaxy on the way, so please subscribe to the feed if you’re interested.