20 years deep into a '2-year' mission: How ESA keeps Cluster flying

Space Extenders ESA's Cluster mission is heading into its third decade of operations. The Register spoke to some of the people behind the four spacecraft about how the team turned a five-year nominal lifetime into 20 years and beyond.

The four identical Cluster spacecraft, flying in a tetrahedral formation, are designed to study structures in the Earth's magnetosphere in three dimensions. The proposal for the mission was made in 1982. By 1996 Cluster was ready to be launched.

Unfortunately (as it turned out) the opportunity to take advantage of a free ride on the first flight of the new Ariane 5 rocket was impossible to resist. The launch on 4 June 1996, carrying all four satellites, ended in failure. The Ariane 5 was destroyed, and the wreckage of years of work on Cluster was scattered over the swamps.

Undeterred, ESA set about building a fifth Cluster spacecraft, using flight spares for a potential launch in 1997 before deciding that, heck, the design and development was done, and a single Cluster spacecraft wouldn't meet all the objectives, why not build more and catch the peak of the solar cycle in 2000?

Cluster (or Cluster II) would use a pair of Soyuz boosters this time rather than an Ariane 5 and the four spacecraft named Rumba, Salsa, Samba and Tango (named by Raymond Cotton, a Bristol-based civil servant from the UK) were successfully launched in 2000 from the launchpad at the Baikonur Cosmodrome.

The mission was initially planned to run for two years, and the nominal lifetime of the spacecraft was between four and five years.

The Cluster spacecraft themselves weighed in at approximately 1,200kg at launch, including 650kg of fuel (most of which was used up just after launch to get the spacecraft to their operational orbits). The cylindrical vehicles (measuring 2.9m in diameter and 1.3m high) were wrapped in six curved solar panels and used five silver-cadmium batteries to keep things ticking over while the spacecraft were in the Earth's shadow.

About those batteries

We spoke to Bruno Sousa, who became operations manager for the mission five years ago, about how the team had kept the four Cluster spacecraft flying for so long.

Unsurprisingly, the batteries eventually became a challenge.

"The batteries," he told us, "were designed to produce low electromagnetic noise, to make sure they wouldn't interfere with the science."

Certainly, for a mission concerned with measuring magnetic fields, the interference from regular powerpacks would not be welcome. As such, the manufacturers came up with something a bit special, but with a slightly limited lifespan.

"It was only supposed to last two years," recalled space plasma physicist, Philippe Escoubet, who has been on the Cluster project since 1993, "but by doing maintenance of the battery, discharging and recharging slowly, we could keep them for five years."

Delighted to see that the spacecraft could survive the plug-pulling, the gang investigated further and found logic lurking within the Cluster satellites that would stop them booting autonomously...

"They were probably designed better," said Sousa. "They lasted actually for 10 years."

"But after five years," said Escoubet, "they started to crack."

"They were cracking and leaking," Sousa explained. The team were able to detect the leakage from the thermal behaviour of the spacecraft and by tiny orbit changes. The latter could only be explained by something leaving the spacecraft "and that was fluid from the batteries," said Sousa (who had yet to join the team.)

At first the team adjusted the power load drawn from the batteries while the spacecraft were in eclipse. Escoubet recalled the introduction of a new mode to the spacecraft, where it could run on minimal power "and listen to commands from the ground".

Since everything eventually had to be turned off in this mode, other than the radio, to preserve the batteries, handling an eclipse with the heaters deactivated presented its own thermal problems.

"We need to preheat the spacecraft before they enter the eclipse, such that they don't get too cold," Sousa explained. Frozen fuel and a damaged radio amplifier would be less than ideal.

"We had a PhD doing some analysis in our thermal behaviour. He came up with some recommendations in terms of how long we needed to heat up and provided a strategy, which we implemented."

"It has worked fine since then."

However, the power situation continued to deteriorate and eventually the team were forced to fully discharge the batteries and keep them at zero capacity to avoid any further leakage or contamination.

"At 10 years," Escoubet said, "it became a mission where we did not have any more battery."

"From this point onwards," Sousa told us, "the spacecraft actually had to go through a full reset." An eclipse meant no Sun on the solar arrays, which in turn meant no power and a shutdown for Cluster.

"The first time it happened," said Sousa, "this was a rather uncontrolled shutdown... the spacecraft were not designed to be switched off."

With little choice in the matter, the team was forced to hope for the best as the spacecraft essentially died in orbit.

Power from the solar arrays eventually brought the vehicles back to life as they emerged from eclipse "and things would start to come up in some random order, which would often leave the spacecraft in an inconsistent state".

Delighted to see that the spacecraft (and more importantly, the radios) could survive the plug-pulling, the gang investigated further and found logic lurking within the Cluster satellites that would stop them booting autonomously.

"This was probably set there during development on the spacecraft so they could control the activation," explained Sousa, "and it was not meant to be used during flight.

"It was only when we started having to switch them off during eclipses that we figured out this logic was implemented there, and we could use it for our benefit."

No booting meant no uncontrolled activation.

"We would then pick up the spacecraft in the blind, start radiating commands and slowly going through the switch on procedure."

It was a rather manual process as the team became accustomed to the new way of working.

"In 2011," said Sousa, "the orbits were low. They were basically having eclipses for 11 months in a row and our orbit has 54 hours of duration for each orbit. So that meant every two days they had an eclipse and they had to do all this cycle of switching it off switching it back on."

2011 brought with it other problems (more on those later) and while the team did manage to get some science done and dump the data before the next eclipse, it was very labour-intensive.

The team slowly developed new tools to automate the process. When they began, "it would require sometimes three to four hours on each spacecraft to get them fully back reconfigured. And it meant for all four spacecraft about 15 hours of work."

Nowadays, Sousa told us, "we have an automated procedure on the ground that starts commanding the spacecraft in the blind and does the recovery of a single spacecraft in about 40 minutes." Parallel operations mean that sometimes all the spacecraft are up and running in under two hours.

Youtube Video

"Now we have been through more than 1,000 eclipses," said Escoubet, "and it has worked all the time."

"With every year we basically learned something," explained Sousa, "and we adapt the next year to do it a bit more efficient."

The 'Dirty Hack' revisited

Escoubet reminded us of another infamous near-death experience that occurred in 2011.

"We realised in 2011," he remembered, "that we could not switch on half of the payload anymore on one spacecraft." The spacecraft in question was Samba.

"It was consuming too much current, and nobody knew what it was, if it was a failure of a component or some programming issue. We tried again, and after a fraction of a second the spacecraft switched off the five instruments."

The five comprised the Wave Experiment Consortium (WEC), used for measuring the electrical and magnetic fields. Losing the payload would have a serious impact on the mission's "four-spacecraft" science.

The Register noted this event at the time: ESA's investigations had shown the five power switches locked in the "closed position", something from which there were no recovery procedures.

"We found tests that were done in 1994," recalled Escoubet, "and realised that the current consumption with the five instruments switched on at the same time would not work.

"They would have to switch on only three at a time and then the last two later on. So, we had the problem that we did not have enough current in the spacecraft to do that."

However, what the team did have was two current lines, of which only one was normally used. If that failed, then the system could switch over. "But then," explained Escoubet,"we got the idea to use both in parallel and get enough current for the five instruments to switch on."

"However, we had this problem that to switch on both powerlines in parallel you needed to send two commands. And the minimum time between these two commands could be only 39 milliseconds."

"We needed 11 milliseconds."

Time to wheel out those hacker skills.

"They decided to hack the system to use only one command but to quickly change the memory the command was using. By changing the memory, they could then switch on the back-up line."

It was indeed a "Dirty Hack" as ESA put it. The team tried it on a healthy spacecraft, and it worked. They then tested it on the spacecraft with the problem without success. After what we suspect were a few deep breaths and five or ten minutes, the team tried again and "it worked."

"It was a clever trick," said Escoubet proudly, before laughing: "maybe more clever than the thing with the batteries!"

"Dirty Hack" or not, the instruments were returned to normal operation.

Running on (near) empty

The original mission plan called for the spacecraft to use separation distances of anywhere from 600km to 20,000km depending on the needs of the science, while the highly elongated orbits could see the spacecraft 119,000km from Earth, and other times considerably closer.

Much of the fuel carried by quartet was expended in reaching their operational orbits and then, according to Sousa "a lot of it was used on the first five years to aggressively change the formations of the spacecraft."

This was all well and good with the original nominal mission length, but as the spacecraft demonstrated their longevity, the team realised that the fuel would soon be exhausted. Sousa reckoned the timeframe was 2005 to 2008.

Once again, the team adapted to cope as Sousa explained: "Instead of doing aggressive formation changes in consecutive orbits, they started doing small drifts." These would slightly change the orbits of the spacecraft in relation to each other. "We would wait about a month until the drift had accumulated long enough and then we would stop it with another small manoeuvre."

Back in 2007, the drift approach saw the Samba and Tango spacecraft achieve a separation distance of just 17km.

"Just pushing or giving a kick of one spacecraft you use maybe a few hundred grams of fuel, but if you wait long enough, it will drift on orbit for a few hours, so we can reach thousands of kilometres or also very small distances, down to three kilometres" explained Escoubet.

The science approach had changed, but the fuel consumption was reduced drastically: "at the moment, we are only spending between 100 and 300 grams of fuel per year," said Sousa.

The best estimate has 6kg of fuel left in the spacecraft (about 1 per cent of the original) although Sousa cautioned that "the uncertainty of that value is in the order of three kilos. So maybe we have three, maybe we have nine."

Enough to see the spacecraft through to their eventual re-entry as the orbits naturally decay.

The end will begin in 2024, when the first of the Cluster quartet enters the Earth's atmosphere. Another will fall in 2025 and the final two will burn up in 2026, more than a quarter of a century after launch.

Growing extra instruments, Mars Express style

As with the Beagle 2 departure camera on Mars Express, the Cluster gang "rediscovered" a camera on the spacecraft late into the mission.

Originally added to snap images of the separation of the Cluster spacecraft after launch (and maybe capture a slice of the Earth), the cameras had lain dormant for 15 years. They had been used successfully on one of the Soyuz launches, but not worked in the other due to a spacecraft reboot in the launcher.

"When I joined the mission and was learning about the spacecraft, I learned there were a couple of cameras.

"The question was, could these cameras actually photograph the earth?"

The answer was a blunt "no". The position of the cameras on the spacecraft meant they were usually pointed at the "sky" rather than directly at the Earth.

"But we looked at it carefully, we went to the user manual to see the details of the cameras, the field of view. And we did some calculations. And we figured that there were some moments in our orbit, where the earth might just pass in the field of view and we would be able to see something."

The team decided to try it out. The camera needed a little commissioning to bring it online and the team ensured it could be swiftly shut down if it consumed too much power or interfered with the rest of the payload.

Once confident the thing was actually working, pictures were snapped and stored in the onboard mass memory.

"And we were actually lucky on the first attempt; we saw on the first test pictures one of our low gain antennas."

"We then had the camera take pictures when we expected the Earth to be nearby and sure enough, we actually got it in the corner of the field of view.

"So, after that we did several campaigns to use the camera, particularly when we were not doing science, like around Eclipse period. We would switch on the camera, take some pictures and play a bit with the with the exposure until we actually got some nice shots from the South Pole upwards. We can see in some pictures we can see Australia and the tip of South Africa under the clouds."

What the #VMCSouthPole images are about... Roughly! pic.twitter.com/NbmnrCPUTZ

— ESA Cluster (@ESA_Cluster) February 18, 2016

The cameras are not used regularly: "it was purely for public relations" laughed Sousa, although we'd contend that repurposing an instrument that had been deactivated for 15 years in space sounds like a lot of engineering fun.

Epilogue

The bonus science from the long-lived mission is outstanding. Escoubet explained that the mission has now been able to observe almost two solar cycles, the approximate 11-year change in the Sun's activity. "This is very important to see trends and observe differences," he said.

"The second aspect," he added, "is that we are now doing collaboration with newer missions." The data from Cluster can be combined with other spacecraft to create multi-point observations and track global changes in the Earth's magnetic field.

The spacecraft remain healthy; Escoubet told us that the team experienced some failures just after launch, but otherwise there had been minimal degradation over the years. A transponder has failed on one spacecraft, but the backup continues to function.

However, the continuation of the mission must be justified. Escoubet told us that the team were currently preparing a case, including new science goals, for the extension to 2023 and 2025.

As for the extraordinary longevity, both Escoubet and Sousa paid tribute to the designers of the spacecraft and the support of the manufacturers. "It's a very strong design," said Escoubet, "not too much flexibility, but very strong design."

"The design and implementation of the spacecraft was outstanding," agreed Sousa, paying tribute to the integrators and the experience gained from the first four spacecraft built.

While the computers on the ground have moved on (Sousa explained that the original VAX VMS hardware was long gone, and controllers like things a little more virtualised nowadays - unexpectedly handy for today's remote working needs) the core interface remains the same, albeit augmented by modern tools.

And the spacecraft? "They are just marvellous pieces of technology", said Sousa proudly.

Cluster continues to be operated by a small team and, should funding permit, carry on performing science until the end finally comes. "We got the ESA award for teamwork excellence in 2014," recalled Escoubet happily.

We'll leave the last word to Sousa, current operations manager of a team that has stretched a two-year mission to 20 years and beyond through ingenuity and innovation:

"Never give up. Never surrender." ®