NASA's Deep Space Communications Network Continued Working Through A Difficult Weekend

As the flames of the raging brush fire dubbed the Station Fire threatened the northern edge of NASA's Jet Propulsion Laboratory on Saturday, Aug. 29, the managers of NASA's Deep Space Network prepared for the worst.

The Deep Space Operations Center at JPL is the nerve center for the Deep Space Network, an international network of antennas that send and receive information to interplanetary spacecraft. Staffed 24/7, 365 days a year, the JPL hub is constantly active connecting three major antenna sites, numerous mission operation centers run by NASA and an international group of space agencies, and more than 30 spacecraft flying throughout our solar system.

"We were more like the nervous center that weekend than the nerve center," said Wayne Sible of JPL, the network's deputy program manager for Deep Space Network development, operations and services.

The Deep Space Network operations managers knew that, fire or no fire, time was critical for sending software programs to and downloading diagnostic information from several spacecraft, including the Mars Reconnaissance Orbiter, which had an unexpected computer reboot the day the fire started on Aug. 26, and the Dawn spacecraft, on its way to the asteroid belt.

The network's antennas that send and receive information to spacecraft, located at Goldstone, in California's Mojave Desert; near Madrid, Spain; and near Canberra, Australia, were never in danger. But Sible and Jim Hodder, the network's operations manager, got word on Friday, Aug. 28, that the Station fire, which started in the San Gabriel Mountains above the Laboratory, was burning towards JPL. Emergency managers and senior JPL administrators called for JPL to be closed, except for essential personnel, on Friday evening.

A flurry of phone calls followed -- to the Deep Space Network team, the mission operation centers and ITT Systems Division, the contractor that provides the operators for the operations center at JPL.

On a phone call with Hodder, the team decided to move network operators to a facility in Monrovia, CA, where other support work is normally conducted for the Deep Space Network. The Monrovia building - about 15 miles from JPL -- offered basic access to the critical systems, though the operators would not be able to use personalized computer scripts or notes that facilitate their work.

Aerial Firefighting Photo Credit John Livzey

It seemed practical, since activating the emergency control center at the Goldstone complex in California's Mojave Desert would be more disruptive and require some suspension of communications while they moved staff 150 miles to that location.

Two of the five Deep Space Network operators on weekend duty were sent to Monrovia, but three volunteered to stay at the control center at JPL, to ensure systems continued to operate normally, to keep connections open with the flight projects, and to maintain the flow of engineering and science data to flight projects and scientists around the globe.

The three who stayed at JPL - along with about 40 other mission-critical personnel at any given time - were told not to spend much time outside. Hodder called frequently to check on the health of the crew and to obtain status reports on the network.

On Saturday afternoon, Sible and Hodder were ready to pull out those remaining three operators and put further communications with the network on hold if the fire reached the Mesa, a flat helipad and testing site at the northern edge of JPL.

Aerial Firefighting File Photo

That afternoon, the fire burned to within an eighth of a mile of the northern border of the lab. Emergency managers told staff to be ready to evacuate in 30 minutes.

With fire department hand crews cutting firebreaks, helicopters and fixed-wing aircraft dropping water and flame retardant, and the wind shifting, the danger passed on Saturday night. An unpleasant haze of smoke settled on the lab, but the air had cleared enough for the network operators in Monrovia to return to JPL Monday evening. The rest of JPL opened as usual on Tuesday morning at 6 a.m.

In the end, the Deep Space Network was able to complete its 182 scheduled uploading and downloading sessions with spacecraft over the weekend without interruption.

"It went very well," Sible said. "Nobody saw any hiccups whatsoever."

FMI: http://deepspace.jpl.nasa.gov/dsn/