Spacecraft Headaches

Two nights ago, the spacecraft I work on had some sort of an anomaly. Two of the instruments "safed" themselves -- put themselves into a "safe" configuration and stopped collecting science data. In the aerospace industry, "safing" events are almost always bad in that they signal that there's something wrong with the spacecraft -- usually very wrong.

In this case, it seems there was some kind of a software process that got stuck in the onboard computer. The hardware for the instruments failed to detect a "heartbeat" from the flight computer, so the instruments decided conditions were not so good and threw in the towel. The remainder of the spacecraft appears to be healthy and is continuing operations normally, but our flight software geeks are all up in arms because they don't know what caused the problem. They're being conservative, which is good, but being conservative will probably translate into shutting down operations for a good long period of time while they puzzle out what caused the problem.

In this case, though, I think they goofed. Today, they suggested that they might want to keep the two safed instruments powered down for several weeks while they looked into the problem, which prompted our Project Scientist (the biggest cheese on the project, next to the project manager) to state, "Well that's basically saying that we've failed our mission success criteria." It was a beautiful moment in that it got the spacecraft guys, who tend to be focused more on the hardware and less on what we're actually doing (collecting science data), to come to solutions faster. You see, one of the instruments that safed is tasked with doing global mapping of the atmosphere of Mars over long periods of time. Missing several weeks of data puts a huge gap in their data set, which will make it impossible to complete their objectives. Sometimes the hardware guys forget that the whole reason we're doing this is for the science data that we collect.

For me, this whole thing means a lot of work. The small team that I manage are all scurrying around trying to figure out what this means for us. We're likely going to be constructing several command products -- most of which won't end up being used. It's an awful lot of work for very little gain, but it is necessary to keep the spacecraft operating.

So much for the quiet weekend I was hoping to have. My wife was planning to buy the newest Harry Potter book and spend the whole weekend reading it while I chased the kids. I hoped to either take them to the beach or to the local amusement park where we have season passes. Now it's likely I'll need to stay home near the phone in case things get out of hand. It's what I get paid for, but it doesn't mean I'm not allowed to grumble. *grumble* Hmm ... maybe I'll send my wife with the kids, and I'll stay home and read Harry Potter!