Comments on: Things That Are Important: Where Clauses

By: k

Mon, 07 Apr 2008 15:11:42 +0000

I remember forgetting the where clause and accidentally blowing away the Audiogalaxy user data table. Luckily we had a backup. That was when there were only like 3000 rows in that table.

Matt, don’t forget we were partying like it was 1999. Since it was, you know, 1999.

By: Tom

Tom — Tue, 25 Mar 2008 17:04:03 +0000

@Chuck: That is certainly a lesson I hoped the responsible developer learned! My point is that from a higher level you need good tools to detect weird problems with your cluster. It may be a careless developer, or it may be a hacker, or a bad script, or a faulty switch, or something entirely unexpected.

By: Chuck

Chuck — Tue, 25 Mar 2008 16:50:39 +0000

What about the obvious moral? You don’t make changes to production databases unless you are fulfilling a business requirement. Temporary hacks are bad enough, but temporary hacks so you can get your game on? Impressively unprofessional.

I like a good game of Q3 as much as the next guy, but there are some things you just don’t do!

By: Tom

Tom — Tue, 25 Mar 2008 14:29:34 +0000

@wac: Yes, transactions would have been nice. But of course, we were on MyISAM. More of a disk backed hashtable than a database, really.

@matt: I guess I should say that for us, it wouldn’t have been practical. Mirroring our production DB would have required about 15 machines, which we simply couldn’t afford. We did have a relatively empty dev database that we used as much as we could, and we certainly didn’t encourage folks to hit the production machines, but sometimes it made life easier.

By: matt

matt — Tue, 25 Mar 2008 14:14:26 +0000

“Don’t let your developers have access to the production database? Maybe, but that isn’t practical for a small startup.”

Why not? That’s like saying that sysadmins should be dabbling in the accounting. Developers should NOT touch the production database. It’s called production for a reason. If they want to look at the data, you can / should have a nightly snapshot back onto a performance database or some such they can look at. Yes, even for a small startup.

By: wac

wac — Tue, 25 Mar 2008 07:08:14 +0000

This is also another case where forcing people to use transactions acts as a sort of insurance policy.

> BEGIN

> UPDATE [something dumb goes here]
“Why’s this taking… oh damn!”
ROLLBACK

By: Tom

Tom — Tue, 25 Mar 2008 06:03:53 +0000

Thinking about it some more, I’m guessing that they ctrl-c’d the query pretty quickly and thought that would prevent the update. We had a good group of folks working there, and I’m pretty sure that if they knew what had happened, they would have let me know. Oh well — it makes for a good story.

By: jack

jack — Tue, 25 Mar 2008 05:33:46 +0000

Jolly good story!

I quite agree that it was careless (irresponsible?) of the unknown developer for making that screwup in the first place, but to not ‘fess up and fix it afterwards? (you: “The system’s going to shit. What changed today?” them: “sound of crickets chirping and wind blowing”) Totally unacceptable.

Like most devs, I don’t mind so much when somebody screws up (everybody does) but let me tell you what, not admitting to it (or worse yet, covering it up) makes the situation 1000% worse because it delays the solution and damages trust which is the foundation of any working team.

Thanks again for sharing an Audiogalaxy “war story.”

By: GarethGilson.com » Blog Archive » mySQL: Require WHERE Clauses

GarethGilson.com » Blog Archive » mySQL: Require WHERE Clauses — Tue, 25 Mar 2008 04:08:27 +0000

[…] just read over a post entitled “Things That Are Important: Where Clauses” (link here), which brought something interesting to my […]

By: seth

seth — Tue, 25 Mar 2008 02:05:44 +0000

great article! thanks for taking the time!