Comments on: Design details of Audiogalaxy.com’s high performance MySQL search engine

By: Adding Search with Lucene | Ajaxonomy

Adding Search with Lucene | Ajaxonomy — Mon, 30 Jun 2008 03:45:44 +0000

[…] need to build my own search engine.This is not really that hard to do. I would recommend you read some articles and then download Managing Gigabytes for Java. Those articles are by Tom from AudioGalaxy. […]

By: Ueberbach JP

Ueberbach JP — Fri, 27 Jun 2008 01:19:09 +0000

I agree on the fact than when you want real speed you should be able to programm in C(++). We have kinda similar structure on mercuriusguides datacenter with a diffence in the webserver cluster; our webserver only outputs xml. The rendering servers which render the xml with xsl to html are located all over the world on (paid) hosting servers. The xml to be rendered is outputted, level 9 compressed, from our datacenter. Just another way to keep costs down (power,cooling,bandwidth,rackspace, etc…) and be more efficient. We have currently about 80000 unique users per day, about 10 million page views per month. Cpu of mysql database always below 5%. Especially the plugin structure of mysql is very usefull to increase performance. Hoerai for that.

By: Large Mysql Database

Large Mysql Database — Sun, 06 Apr 2008 15:47:44 +0000

found your site on del.icio.us today and really liked it.. i bookmarked it and will be back to check it out some more later ..

By: Tom

Tom — Sat, 15 Mar 2008 17:48:02 +0000

@Vijay: Thanks for the pointer — I’ll dig through the Nutch code and see what I can find.

By: Vijay Chakravarthy

Vijay Chakravarthy — Sat, 15 Mar 2008 02:06:45 +0000

Regarding the indexing for situations where the dictionary doesnt fit into memory, basically a standard approach is to index every nth entry of the dictionary, and then have a prefix based compressed representation of the n+1 to 2n th terms of the dictionary. Obviously this adds another disk seek to get the postings, but it works ok in practice especially if you cache the postings.

I would look at some of the nutch source code, since it has scaled pretty well for large internet searches.

By: Lessons Learned Scaling the Audiogalaxy Search Engine | Spiteful.com

Lessons Learned Scaling the Audiogalaxy Search Engine | Spiteful.com — Sun, 09 Mar 2008 03:45:12 +0000

[…] About « Design details of Audiogalaxy.com’s high performance MySQL search engine […]

By: Tom

Tom — Thu, 06 Mar 2008 20:42:21 +0000

@Jonathan: Hey — good to hear from you. I won’t call it a problem, but that was certainly a characteristic of AG (and possibly most start ups): stuff really wasn’t documented. In the time it took me to write that article I could have knocked out a few new features.

By: Jonathan

Jonathan — Thu, 06 Mar 2008 15:27:14 +0000

Nice write up Tom. I think I learned more about the design details of AG from this post than from working for AG for over a year.

By: Tom

Tom — Tue, 04 Mar 2008 17:16:11 +0000

I recently read through this and a few of your other posts. It’s always fun reading how others create hugely scalable systems.

“The lengths people will go instead of just using a scaleable PostgreSQL server…”

Haha, this was my thought! Although managing to scale a PHP/MySQL solution to 70 million searches per day is quite amazing and certainly explains the need for a completely custom-coded C system and transfer protocol. A technological achievement to be sure.

By: Uri L.

Uri L. — Sun, 02 Mar 2008 14:07:59 +0000

thanks for sharing, was a great reading.