It was July 4th yesterday. Fireworks. I didn’t go down to the waterfront to see them in San Francisco this year, I was in a “place” (that might possibly have served beer) having fun. But it reminded me of this a couple of years ago. On July 4th 2012 the San Diego fireworks display, one of the biggest in the world, detonated simultaneously. They fired every single thing at once, or within about 10 seconds. Luckily their protocols were such that everyone was locked down inside bunkers and nobody was hurt even in that worst case scenario (which involves 5 barges spread over nearly 15 miles).
Of course it was a PR disaster. But let’s face it, for everyone who saw it, it will be the most memorable fireworks show of their lives. The press had a field day about “massive hardware failure” but if you are in technology (and if you are not, what the hell are you doing here?) that sounded dodgy. A big hardware failure would result in no fireworks going off at all. Totally believable. But all at once? That has to be a software screwup. And so it proved. A few days later they issued the postmortem. It reads a little like the postmortem on chips that need a respin, everything was checked but somehow something got through. We used the wrong CPF file. The old version of the IP. Whatever.
The report on the fireworks is here (pdf).
Warning: heavy geek factor if you read further!
This reminds me of another screwup when a space vehicle failed to get into orbit around Mars (I think, I’m remembering all this so the details may be slightly different but the basic idea is accurate).
The code was written in Fortran which, unless you are of a certain age, you have probably never used. It was invented in the 1950s when a lot of what we know now…well, we didn’t know. It had a number of convenient features which people though were neat like:
- variable names could have spaces in them (that were ignored, in fact all spaces were ignored except in quoted strings). So you could say R AND R (we didn’t have lower case back then). Or, as turned out to be significant, DO 13 I
- variables didn’t have to be declared. if you used one then if it started with I-N it was integer and otherwise was floating point. DO13I is thus a floating point variable.
- Loops in Fortran were known as DO loops and had a syntax like this:
[INDENT=2]DO 13 I=1,5
which meant execute all the instructions between here and the label 13 with I taking the values from 1 to 5, like a for loop in C. No we didn’t have brackets {} either. Just labels.
- Loops were ended with a labelled statement. So that you could GOTO the end of the loop (equivalent to break in C) you could label a CONTINUE statement that did nothing. Whereas jumping to the last statement of a loop if you labelled it would execute that statement. But there was no problem having a CONTINUE statement not in a loop. You wouldn’t get a peep out of the compiler.
So what happened to the spacecraft? Someone put in a typo:DO 13 I = 1.12
with a period instead of a comma. It is all totally syntactically correct. It was a Newton-Raphson iteration to solve a differential equation to calculate the amount of speed required. But with the rules I told you about this was actually:DO13I = 1.12
Namely declare a floating point variable DO13I and assign it the value 1.12 (and never use it again. No we didn’t have Lint type tools back then either. Or tools period).
So instead of the loop running 12 times (to converge on a good value) it ran through once (since it wasn’t a loop, just an assignment statement) and it used the initial approximation. Newton-Raphson is so efficient that it doesn’t much matter what the initial approximation is in most circumstances, nobody makes any effort to get it close. Just use 1. And maybe a couple more iterations if necessary (and yes, I know about double roots, but that is nothing to do with this).
Result. Loss of spacecraft. One character wrong.
Back to fireworks. This video has gone viral so you have probably seen it. A guy flys a quadracopter drone through a fireworks display with an HD GoPro camera. It survives (although takes a few minor hits and lots of stuff flying by). Happy Day-After-Independence-Day
P.S. Someone a decade ago asked me what the Brits do on July 4th. “Nothing, we lost” I replied.
More articles by Paul McLellan…
Next Generation of Systems Design at Siemens