In early 2003 Actel announced a new product family: RTSX-A. It was a family of antifuse FPGAs aimed at the satellite market. Customers had known for a long time that it was coming and there had been prototypes available for many months. Our space customers loved the product. This was going to be a big win for us! One of the first programs to use the product was the Mars Rover program. There were four Mars Rovers made: two went to Mars and two lived in a huge sandbox in Pasadena which allowed scientists to emulate conditions on Mars. The two that went to Mars were named the Spirit and the Opportunity. NASA’s plans were that the Rovers would live for three months before succumbing to the treacherous Mars environment. The Spirit launch was on June 10, 2003. It would take the Spirit about eight months to make the trip to Mars. The Opportunity was launched a month later.
Shortly after the launch we got reports from potential customers of a few RTSX-A burn-in failures in their labs. What’s burn-in? It’s a test that assures the user that a part that starts out good will stay that way after being in actual use for some time. Testing parts for one week at extremely high temperatures is the normal way to assure that they’ll last a long time at normal temperatures. We hadn’t seen failures with our first tests, but when we got those reports from our customers, we looked harder. When we looked harder, we saw some failures on occasion. We would burn-in around 100 units at a time. Sometimes we would get 1 or 2 failures. Sometimes none.
A small number of failures is worse than it sounds! A typical satellite would cost in the neighborhood of 100 million dollars in those days. The Mars Rover project cost much, much more than that. The Mars Rover project used the RTSX-A. Uh oh!!
Satellites can’t be repaired. If one IC fails, the cost of the entire project may well be flushed down the toilet. Worse, there wasn’t just one RTSX-A part in each Rover. There were, as I recall, 38. That meant if there was a 1% chance that one particular Actel part would fail, there was a 38% chance that one of our parts somewhere in the Rover would fail. (My math isn’t quite correct here, but you get the point) It was clear something wasn’t quite right, but we couldn’t figure out what.
The Mars Rover program was by no means the only program planning to use our parts. There were many others! Word got around the space community. Many customers weren’t sure if they dared to launch their satellites. They looked to us to tell them it was OK. We couldn’t. We just didn’t know. We couldn’t figure it out. A few skeptics thought we were covering something up. We weren’t. It was a very tricky problem. We were working hard on it, but we just didn’t understand what was going on.
Then, I got a call from Bill Ballhaus, the CEO of Aerospace Corporation. Aerospace Corporation operates a federally funded research and development center. They provide technical guidance and advice on all aspects of space missions to military, civil, and commercial customers. Dr Ballhaus asked me to come to the Aerospace headquarters in El Segundo to discuss “the reliability problem with Actel FPGAs”
I, of course, accepted the invitation, but if you had offered me a choice of going to this meeting or getting a root canal on every tooth, my oral surgeon would be a richer man today. The invitation appeared to be for a one-on-one meeting between me and Dr Ballhaus. I planned on going by myself, but our VP of Technology, Esmat Hamdy, saw it differently. He didn’t quite trust my technical savvy. He thought that, if the meeting turned out to have a lot of technical content, I might not be able to answer all the questions well. So — Esmat insisted on coming with me. Bless his heart!
We flew to LAX and then took a quick cab ride to Aerospace. It’s about a mile from the airport. A secretary led us to a conference room — except it was more like a sports arena. There was a long rectangular table that probably sat 15-20 people. Then there was an aisle circling those seats. But on the other side of the aisle, there was an elevated set of chairs circling the table below. There were maybe another 20-25 chairs in that set. In total I would guess 30 or 40 chairs. All but four were full. Not full of just anybody — but full of PhDs. Full of experts in any aspect of integrated circuits that you could think of. Full of technical wizards who all had at least double my IQ. One of the empty chairs was for Dr Ballhaus. One for Esmat. One for me. We took our chairs and were ready to start the meeting but Dr Ballhaus said that we’d have to wait. The last chair was for a high ranking Air Force general who had invited himself to the meeting. The general was running late. When he got there, there was plenty of bowing and scraping done. He was the head honcho!!
They hammered into me just how important this was. The Rover program cost about one billion dollars. The Rovers had been launched. There would be no calling them back. No fixing them. If they went bad, that would be a billion dollars flushed. And worse, the Mars Rover project was by no means the only satellite project with plans to use Actel. There were several military satellite programs related to our national defense as well. Those folks were even more worried than the Mars Rover people. They were not happy campers!!! As you would expect — the general had zero interest in putting up military satellites critical to the nation’s defense that were likely to fail!!!!! (That unhappiness earned me an invitation to meet later with Peter Teets, the Undersecretary of the Airforce, in the most secure area of the Pentagon. Mr Teets wasn’t a happy camper either, but that’s another story.)
Back in Mountain View, we were breaking our picks. We didn’t always see failures. When we saw them, there weren’t many. That makes the problem harder to solve. We suspected what we called the programming algorithm. The oversimplified explanation of programming an antifuse is this — put a high voltage across it, the dielectric will rupture and the antifuse will be a conductor for the rest of time. — In fact, it’s much trickier than that. There are a lot of knobs to turn. How high should the voltage be? How much current should flow through the fuse? How many times should you repeat what you’ve done? How long should you apply the voltage? How much should you “soak” the fuse? We would twist some of these knobs, come up with a new algorithm, and voila. No failures!!! Problem solved, right? Wrong!!! The next time we’d run exactly the same test, there would be one or two failures. We were completely perplexed!
The first Rover (the Spirit) was launched prior to suspicions that we might have a reliability problem. Then came the reliability worries. And then, around New Years when the reliability concerns had become rampant, the Spirit reached Mars. It was a big deal in the press.
The Spirit Lands on Mars!
It’s working fine!!
It was on the front page of every newspaper. Boy. Did we ever feel good. The Spirit was working perfectly!!! …..…… But then — one week later —-
The Spirit Fails!!
The Spirit went bad. That was on every front page too, but in bigger letters. Was it the Actel parts? We didn’t know, but in my judgement … it could well have been. I was terrified! I could picture the headlines when it was determined that Spirit failed because of an Actel part. I could picture the lawyers lining up to file lawsuits against us. I could picture the process servers skulking in the bushes waiting to spring out and serve me with subpoenas. It wasn’t pretty!! In fact, it was really, really ugly!! When I went home after work that night, I opened a bottle of wine and drank the whole thing by myself. I like wine …… but not a whole bottle. My advice? — Don’t do that!!! It didn’t work out well!!
Luckily, I was wrong. The Actel parts were not at fault. After a week or so, NASA figured it out. It was a software problem that was fixable by uploading new software. They fixed the Rover, and independently we tracked down our problem and fixed it. To my knowledge Actel (Now part of Microchip) has never experienced a failure in space.
Scientists had planned for the Rovers to live for three months. When did they actually die? The Spirit lived for 6 years. The Opportunity 14.
Epilogue
Somewhere around the year 2000 our board of directors asked the question, “John, how long do you plan to stay on as CEO?”. My answer: “I’ll retire no sooner than my 65th birthday and no later than my 66th”. When you’re 55, 65 seems really old, doesn’t it? Well, ten years later along came my 65th birthday. January 20, 2010. Funny thing. By then 65 didn’t seem so old. Still – all in all it was the right thing to do. We released an 8K (that’s the document that public companies use to disclose relevant information.) saying that we were beginning a search for a new CEO immediately and that we planned to complete the search and appoint a new CEO within a year.
We spent several months on the search. There were some ups and downs – but bottom line, we made zero progress. We were back to square one. Then came a surprise. We received an unsolicited and unexpected offer to buy us out from Microsemi — an Orange County based semiconductor company that I was barely aware of. We discussed it at length!
Why would we be interested in selling? We were in the middle of trying to transform ourselves from an antifuse company to a flash company. I firmly believed that it could be done (In fact, it was done!!! See my week # 16, “From AMD to Actel”), but it was obvious that it was going to take a long time before we finally started to see the results on our bottom line. The bottom line matters!! Shareholders want high stock prices!! And — for a company our age, the stock price is determined by the bottom line.
Dan McCranie (Who we had recently appointed chairman of the board) went to Orange County and met with the Microsemi CEO, James Peterson. After some negotiations, Peterson offered a price that was higher than we believed we would be able to command consistently in the foreseeable future. After a few vigorous board discussions, we decided that we owed it to our shareholders to accept the offer. We did. On November 3, 2010 Actel became part of Microsemi Corporation and I rode off into the high tech sunset – unemployed for the first time in 45 years.
This is the last episode in the series. I hope you’ve enjoyed reading them as much as I’ve enjoyed writing them.
See the entire John East series HERE.
# Mars Rovers, Spirit, Opportunity, Actel, Microchip, Aerospace Corporation, Bill Ballhaus, Esmat Hamdy Microsemi, Dan McCranie
Share this post via:
Comments
8 Replies to “Crashing the Mars Rovers!!! Actel and Aerospace Corp”
You must register or log in to view/post comments.