« No Credit Crisis, Thanks. We're Australian. | Main | BofA Gives Details on Loan Modification Program »

Latest Stories

• Challenger and the Trouble with Value-At-Risk
• No Credit Crisis, Thanks. We're Australian.
• The Promise and Peril of Comprehensive Deposit Insurance
• Modern Subprime Finance, 101
• Sneak Peek at Weekend Reading

October 6, 2008

Challenger and the Trouble with Value-At-Risk

Lots of people are newly smart and chattering away about the essential stupidity of value-at-risk modeling. Oh, those dumb financial engineers, they keep saying, How stupid do you have to be to imagine that the real world would conform to a mere formula? Ho-ho.

The trouble is, value-at-risk -- a statistical measure of financial losses your portfolio faces given various holdings and market conditions -- actually worked well for some time. It isn't that it was dopey and didn't work; it's that it made some logical and theoretical sense and approximated the real world fairly well -- right up until it stopped working.

We need to stop pretending to be so smart now, and think more clearly about what we propose to do to avoid repeating the mistakes of the past. And I hope the answer isn't that we'll do no more risk modeling, because that's dumb; and I also hope the answer isn't that we plan to get much smarter about how to measure risk, because that's naive (and an academic project). Instead, we need to think about the relationships that do work,  and understand when they stop working, and what to do then (other than panic).

Put differently, we need to understand the boundary conditions of what we know. We have a financial data set, one bounded in both time and circumstances, and when we venture outside it nothing can happen (we get lucky and/or our model is more useful than we knew), or bad things can happen.

Sociologist Diane Vaughan had a great expression for what happens when you don't know what you don't know, and you creep outside the boundary conditions without failing. She called it "normalization of deviance", the business of convincing yourself that what you used to think was outside safe conditions, was actually okay because you got away with it last time.

It's sort of like what happened with the launch of the Space Shuttle Challenger. The launch team didn't adequately understand how far outside design conditions they were taking the shuttle during that cold, early morning launch, and it led to disaster. It depends, in part, on how you look at the data, as the following two figures show. The first presents the data on chronological form, which masks that we are exiting our boundaries into bad places, while the second figure shows the problem clearly.

1) A chronological plot of shuttle o-ring damage by launch date:

2) A plot of temperature vs damage to shuttle o-rings:

Sphere It   |   Digg it   |   Bookmark it   |   Stumble it   |   Facebook it