Sometimes I wish there was some sort of independent wiki or resource that documented how usable projects are. Given how proven they are, something like Apache or PHP or memcached are probably safe to use. But for newer things, it can be hard to find examples of other people using them. Benchmarks are nice, but I want to hear real world stories about how they performed under real world failures — DNS problems, hard drive failures, bad memory, etc.

I have to say that from my experience MySQL cluster is to be avoided as it’s horrifically flakey. Whilst it is meant to be scalable and robust, having tried running it in a large production environment (to store metadata about cache items in memcache) it has a number of massive problems:

. Tendency to core dump, usually at the worst times. And not just part of your cluster core dumps, it *all* core dumps. (e.g. 2 people saying TRUNCATE TABLE for the same table at the same time == coredump)

. Never frees space. Deleting rows just marks them dead, it doesn’t actually free them. So unless your data set / key values are constant then you’re stuffed. This is similar to PostgreSQL, except MySQL cluster *does not* have a VACUUM which frees the space. This obviously compounds the problem above - when your DB is about to get full, a couple of your admins get paged, and truncate a no longer used table.. Bang goes your cluster.

. If your cluster core-dumps, and you need to re-load it from disk, this takes *AGES* (e.g. > 30mins for ~ 4G of data across 4 machines)

. If you don’t re-load the cluster from disk, but just nuke it and start again - your web servers start issuing queries against tables which no longer exist (until you re-create them). This brings the cluster to a grinding halt. About 10 sessions saying ‘SELECT * FROM non_existant_table’ or generally any query to ‘non_existant_table’ is *more than enough* to make the cluster unuseable (>3 mins for a new connection).

As our cache system had to deal with all of this flakyness, we ended up with MySQL being an *optional* component (needed for performance, but if not there would not block requests - however if it wasn’t there cache clears had to be pretty shotgun).. We moved to just using a load of standalone innoDB instances, so that any one could fall over, and only affect a portion of our client-base, as it wouldn’t drag all the others down with it..

If we’d known that MySQL Cluster was so unreliable / untrustable in advance, we never have used it and/or designed our cache system in a different way.

It’s a shame, as it’s a good looking project, and if it’s failure modes weren’t so abysmal, it’d be awesome, but I won’t in any way trust it again until I hear of someone actually using it in production at a reasonable scale (outside of mysql’s marketing literature).

Damn, those were good days. I remember downloading 30 Temptations songs on Good Friday. Over dialup, too.
I wonder if my parents figured out why they got no calls that day…oh well..

- Sue.