Comments on: Crashing When Something Feels Wrong

By: Pat

Pat — Tue, 15 Apr 2008 21:42:01 +0000

All this logging and analysis, in a production service, sounds like it would cost a lot of CPU time and could easily be overkill.

By: James

James — Tue, 15 Apr 2008 13:27:56 +0000

My suggestion is logging combined with log-analysis tools. You *could* monitor processes (like Erlang style concurrrency) as well.

With logs, you just output your info into a file (or db table) and then periodically analyze the data for patterns. These patterns could be trained (e.g. Bayesian filtering) or matched against known problem sequences (e.g. a state machine that acts like a code/regexp parser).

By: john

john — Tue, 15 Apr 2008 11:19:53 +0000

I once ran a messaging system which received about a million records a day. I also ran a document generation system that FTP’d anywhere from 10,000 to 60,000 files a night. Both systems had multiple, custom built monitors which could page developers if problems arose.

By: autotrader

autotrader — Tue, 15 Apr 2008 07:46:33 +0000

Your approach to probabilistic assertions is very interesting. I am using a similar approach on some of my trading algorithms. Basically, if some statistical conditions show that “shouldn’t be there” the system gets ready to dump the trade. However, to make it perfect, I think the system should keep track of every time the “feeling” was correct (i.e., the system did crash or the stock, in my case, moved in the wrong direction). After that, we would need a Bayesian calculation of probabilities before crashing the system.
I haven’t included in my systems because it is too much work, but I still think that is something I will do in the future.
Maybe you may move your system one step forward and record the “feeling” hits and create a simpler algorithm for actual crashing based on that.
At the beginning you would never crash the system, but as you get more data you would crash the system more selectively.
My 2 cents

By: elias

elias — Tue, 15 Apr 2008 06:17:40 +0000

About generating the core dump without crashing with a sigsegv, you could use Google’s Core Dumper
(https://sourceforge.net/projects/goog-coredumper/)

“A neat tool for creating GDB readable coredumps from multithreaded applications — while the program is running.”

By: Emre

Emre — Tue, 15 Apr 2008 05:38:35 +0000

You might enjoy reading this paper, “Pip: Detecting the Unexpected in Distributed Systems”. Among other things, the Pip system supports the kind of aggregate assertions/checks that you mention.

There’s source available at http://issg.cs.duke.edu/pip/

http://issg.cs.duke.edu/pip/nsdi06preprint.pdf
(from the abstract)
We present Pip, an infrastructure for comparing
actual behavior and expected behavior to expose structural
errors and performance problems in distributed systems.
Pip allows programmers to express, in a declarative
language, expectations about the system’s communications
structure, timing, and resource consumption. Pip
includes system instrumentation and annotation tools to
log actual system behavior, and visualization and query
tools for exploring expected and unexpected behavior.

By: JS

JS — Tue, 15 Apr 2008 04:58:02 +0000

Hello Tom,

My bank calls me whenever they see ’suspicious’ activity on my accounts such as a flurry of purchases or other transactions that doesn’t fit in with the normal usage patterns. Airport security in some places are moving toward behavioral profiling, where they watch for people whose actions in the airport or other details doesn’t fit in with the expected pattern are identified for additional inspection. What you’re proposing happens a lot in the real world so it makes sense that it would work for software management as well! Great idea; good luck with its implementation.

By: Steve Cooper

Steve Cooper — Mon, 14 Apr 2008 23:08:06 +0000

Could another type of assertion use spam-filter-like algorithms to detect abnormal (’spam’) situations? Train a filter to recognise odd situations.

By: Adam Blinkinsop

Adam Blinkinsop — Mon, 14 Apr 2008 22:20:42 +0000

Yay for automated monitoring software. Nagios (http://www.nagios.org/) does this for networks (and is extendable for some other things). At my old job we used Hobbit (http://hobbitmon.sourceforge.net/) to watch our Java server instances (memory usage, etc.). There’s no reason why these monitoring programs couldn’t be used to monitor internal program statistics, as long as those stats were made available.

Generally you monitor from your internal network, and then provide some hook for the monitor to get information that’s only accessible from there. (SSH or a limited-access URL, etc.)

Monitoring programs are super-powerful and generally complex. Check them out — it’s a good skill to have when working with production software.

By: Adrian Kuhn

Adrian Kuhn — Mon, 14 Apr 2008 21:06:48 +0000

> I even like running production services with assertions turned on.

The you might like this little Javac plugin, that turns all assertions into explicit exception throws, such that you can not turn them off any more.

http://smallwiki.unibe.ch/adriankuhn/javacompiler/forceassertions/