Starliner snafu could've been worse: Software errors plague Boeing's Calamity Capsule

Troubled aerospace giant Boeing will "re-verify" the flight software code for its calamity capsule, the CST-100 Starliner, after it was revealed that December's anomaly could have been a lot, lot worse.

Boeing had already coughed to a timer error that made the spacecraft's internal clock 11 hours out of whack while sat on the Atlas V. The result was that the Starliner managed to burn through its attitude control fuel on a fruitless attempt to dock with the International Space Station and endure the indignity of an earlier-than-planned return to Earth.

It was unclear if NASA would insist Boeing repeat the flight. The mutterings from the duo seemed to indicate that maybe, just maybe, the capsule had done enough to prove it was safe.

However, a clue that all may have not been well could be seen in between the hand-wringing over the 737 MAX in Boeing's last set of financials (PDF). In the documents, the company recorded a charge to provision for an additional uncrewed mission if needed.

Things were ratcheted up a notch last week after NASA's Aerospace Safety Advisory Panel revealed the timer hadn't been the only thing to go wrong on the Calamity Capsule. A second software error could have led to, in NASA's words, "spacecraft loss".

NASA and Boeing both scrambled to respond, and by Friday the agency shared the initial findings. It won't be easy reading for Boeing, already under pressure for problems elsewhere in the business.

But there's more

Firstly, that timer wasn't the only software glitch. The Service Module (SM) Disposal Sequence was incorrectly translated into the SM Integrated Propulsion Controller (IPC). The result was that rather than performing a burn to dispose of the SM prior to re-entry, the bug could have actually sent the SM bouncing off the Crew Module.

Fortunately, the team noticed that second error while reviewing the code following the first, and uploaded the fix prior to landing.

The third issue concerned the space-to-ground link "which impeded the Flight Control team's ability to command and control the vehicle".

Loss of vehicle

While the former two issues have been fixed, NASA has demanded the team perform "an analysis of whether the issues were indicative of weak internal software processes or failure in applying those processes".

The agency went on to note: "Ground intervention prevented loss of vehicle in both cases."

Bugs in complex software have always been expected. However, NASA gave the blade a twist by pointing out that "there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects."

For its part, Boeing accepted the suggestions of the Independent Review Team and Aerospace Safety Advisory Panel and had already got cracking on "re-verifying flight software code."

It also noted the SM thruster problems, which it put down to a "valve mapping software issue" potentially resulting in "an incorrect thruster separation and disposal burn".

The company delicately described what might have happened, should the fix have not been uploaded, as "unclear", which is not quite as alarming as NASA's "risk of spacecraft loss."

As a reminder, Boeing was picked as the prime contractor for the core stage of NASA's monstrously delayed monster rocket, the Space Launch System, which has been gearing up for testing ahead of a first launch in 2021.

The company would not speculate about when the Starliner might be launched again. It seems Boeing has quite a bit of work ahead as NASA also revealed plans to turn a magnifying glass on the company's culture via an Organisational Safety Assessment (OSA), following an earlier "more limited review of the company".

Competitor SpaceX has already had the OSA treatment, coincidentally after CEO Elon Musk was seen kicking back with a fat one and a shot of whiskey.

After the horrors revealed by the 737 MAX fiasco, we'll be watching through our fingers to see what is lurking in Boeing's workplace culture that resulted in "breakdowns in the test and verification phase failed to identify the defects preflight despite their detectability". ®