The Last Three Days

On the Mars Reconnaissance Orbiter (MRO) project, we have a UHF radio that can communicate with landers and rovers on the surface of Mars. On its way to Mars right now is a lander called Phoenix which is headed for the north pole of Mars.

These past few days we've been in the middle of a major training exercise to help the Phoenix project learn how to operate the lander when it gets there. Since the Phoenix spacecraft has no means of communicating directly to Earth, all the information that passes to and from it must pass through one of the orbiters that are at Mars right now. That includes MRO and Mars Odyssey (another spacecraft I worked on when it was first launched back in 2001).

To that end, coordinating which orbiter is talking to which lander (and the Spirit and Opportunity rovers are likely to still be alive when Phoenix gets there next May) and when is a complex planning problem. In addition, ensuring that the different UHF radios are going to function as advertised, at the right times, can be a difficult challenge to manage.

For me, I'm in charge of all-things-"relay" on MRO. I am coordinating the MRO activities during this training exercise, which has been time- and attention-consuming. My wife and children have been very understanding, particularly since I've been working every evening for the past three days.

Anyway, I thought it might be interesting to record the reports from the last three days. This first one is from Monday:

All,

I wanted to provide a report on the status of the ongoing Relay Surface ORT.

Many thanks go out to Bill <> and company for getting the simulation up and running this morning. The testbed engineers were able to synchronize the clocks within the various testbeds in Denver within seconds of each other. MRO's first overflight of PHX got started up at roughly 7:25 this morning Pacific, with a conclusion of the overflight at about 7:50. At the time, there was no telemetry flowing into the JPL MSA due to issues with the query servers; this was later resolved before the next overflight, which got started at roughly 9:20, with the overflight terminating before 9:45. These first two overflights showed successful data flow in both directions, with the forward-link product (a Phoenix command loss timer reset command) successfully transmitted in both cases (at least that's what the telemetry indicates, though we're still awaiting positive confirmation from Phoenix).

On the first overflight, we returned 2102197 bytes of data, which PHX says is about 2/3rds fill frames. On the second overflight, they "burped" (can't we find a different word?!) 108074 bytes of data -- it is unknown at this time how much of this was fill frames. Electra and MCS appear to have functioned nominally in both cases in response to the Relay NIPCs constructed from the inputs provided by PHX.

The ITL is clocking out exactly as expected, and I witnessed the third overflight of the day (182_04) begin execution within 1 second of the expected time just minutes ago. The background sequence is also behaving normally, and data is now flowing everywhere it should.

Earlier in the day, there was trouble with the Electra "mega"-query script, as it was not correctly extracting telemetry. This was resolved later in the day (still not sure how), though I do believe that some updates are expected to be made this evening by Ricardo Mendoza to clean up some other issues with the script. The "scorecard" script functioned as designed, but had some oddities that confused some people on the other projects (it reports first lock time and last lock time, not total time in lock; it also reports all forward-link frames, not just those that contain forward-link data) -- these are expected to be cleaned up, if not before the end of the ORT, then at least before the next one.

There have been issues with the testbed DOM going into the ORT, which appear to have been resolved. It was decided a few weeks ago that flight data would be mirrored to the testbed DOM in order to allow the ground processes the opportunity to be tested in a more flight-like manner while it was processing Electra data. This had the unfortunate side-effect of filling the available space in the testbed DOM to 97% before it was shut off (in less than a week of being turned on! -- that's how much flight data MRO produces in an easy week!). This could have brought the ground system portion of the ORT crashing down, but this morning, with the flight data flow halted and the HiRISE images purged from the testbed DOM, the used space was brought down to 64% (which is currently down to 50% after some more cleanup). In the midst of all this, the first return-link product came back right as expected, being available for PHX to extract from the DOM at 7:51 this morning. However, the second return-link product was delayed significantly until 11:33 am, but PHX didn't even notice, due to their own issues about which I'm not fully informed.

At 1 pm this afternoon, there was a tagup with the other projects where it was learned that PHX had plenty of GDS issues of their own, but they, too, have been working through them. I do not know for sure, but it seems that their tactical timeline was broken today. Odyssey also had a glitch in their testbed, which required them to restart, and they ended up not supporting a "burp" that PHX had planned -- they were not concerned.

In general, the MRO testbed and associated command products have performed (apparently) flawlessly. Robin <> and all those whom she has called today (you know who you are!) have been invaluable in keeping the GDS running today, as well. Everything seems in order at the present time.

The third overflight has now concluded and there was no data in the link (this was unexpected), and the prox-1 product was tossed. This appears to be a PHX problem, and they're working it now.

Before the night is out, there should be one more "burp" and one overflight that PHX isn't expected to respond to. Tomorrow morning, we will have three sessions where MRO will hail, with only the middle one (at ~8 am) being responded to by PHX -- this will be their primary download for the next day's tactical planning cycle.

At 9 am Pacific tomorrow morning, there will a Short-Range Relay Coordination Meeting where we will discuss some of the lessons that were learned in the past few weeks during the planning portion of this ORT. At 1 pm, there will be another tagup, where we should hear the report from the RDE folks regarding how well the overflights from the past day have gone. I will hold a quick status meeting at 3 pm for MRO personnel.

Sometime tomorrow (mid-day?), it is expected that PHX will be providing us with an update to change the parameters for one of the overflights that are scheduled tomorrow evening. This will need to be turned around by MPST into a new Relay NIPC to be loaded to the testbed. This should be the extent of all further commanding required into the MRO testbed through to the completion of the simulated portion of the ORT. I plan to work these personally.

I will provide another status report tomorrow evening. Good night!

-Roy

There's a lot of acronyms in there, with plenty of other stuff that I won't even bother to try to explain. And while it's not strictly rocket science, they pay me well to understand this stuff.

This next one is from yesterday (Tuesday):

All,

This past day has been fairly uneventful. The OTB continues to function well, as does the GDS -- flowing all data where it is expected. The overflights from the past 24 hours looked like the following:

Overnight and this morning:
-- 182_04 - No data was received. This was unexpected and is being investigated by both the Electra team and by PHX.
-- 182_05 - Nominal overflight, with a small amount of data returned. All went as expected.
-- 182_10 - MRO initiated a hail, but PHX did not respond. This was expected.
-- 182_11 - Nominal overflight, with a small amount of data returned. All went as expected -- this was PHXs "PM" pass.
-- 182_12 - MRO initiated a hail, but PHX did not respond. This was expected.

The first overflight from tonight looks like:
-- 183_05 - Nominal overflight, with a small amount of data expected. It clocked out, as expected, and a small prox-1 data product (5096 bytes) was produced.

Other overflights from tonight should look like the following:
-- 183_06 - Nominal overflight with a small roll towards PHX, with a small amount of data expected.
-- 183_07 - Nominal overflight with a large roll towards PHX, with radiometric data collection included.

Last overflights tomorrow morning and afternoon should look like the following:
-- 183_11 - Nominal overflight. It is unknown whether or not PHX will reply to the hail.
-- 183_12 - Nominal overflight. PHX has made a "tactical" change to bump up the return-link data rate to 128.
-- 183_13 - Nominal overflight, with a small amount of data expected.
-- 184_05 - Nominal overflight, with a small amount of data expected.

Regarding overflight 183_12, PHX originally was anticipating bumping up the data rate for 183_07 to 128 kbps, but after considering that past testing has shown that 8/128 links with radiometric enabled has had poor success when forwarding command products, it was determined by PHX to keep 183_07 unchanged, and instead to bump up the data rate on the 183_12 overflight to 128 kbps. PHX has an "idiosyncracies" list they are keeping on which this will be recorded. For the record, the time it took to process the updated Request APF into the single Relay NIPC SCMF (and have it on the DOM) was 22 minutes; additional files would no doubt have taken more time, and doing the whole thing on the flight side would have taken less time.

It is anticipated that the 183_07 overflight will include their forward-link command load for tomorrow, and it seems likely they will provide it to MRO just before the DDUT of 9:45 pm Pacific. I will be watching this from home this evening and will ensure it gets to the testbed.

At the 1pm tagup today, the RDE statistics were discussed. The results generated by the updated/corrected "scorecard" script were noticeable in the last overflight recorded. All seemed well. Phoenix will provide positive confirmation that PHX did indeed receive the command loss timer reset commands during each overflight in which we closed a link.

For meetings tomorrow, MURO at 9am Pacific has been cancelled, which leaves only the 1pm Pacific Relay Tagup, where we should hear the report from the RDE folks regarding how well the overflights from the past day have gone.

The simulated portion of this ORT is scheduled to end at simulation time 184T15:00 (which is 2am Pacific Thursday morning), however, MRO's last supported overflight concludes at roughly 7pm Pacific tomorrow evening, with data expected on the "ground" within the hour. I can't think of any reason why it would be necessary to keep the OTB up and running once the data is confirmed received, but we will re-evaluate tomorrow.

-Roy

Things have been going fairly well, but the radio geeks found a glitch in their protocols which could be a big deal ... or not. In any case, the powers that be think that there's good to be derived from stretching the simulation portion of the test through tomorrow mid-day.

The following is what I put out tonight:

All,

It would be nice to say this past day has been uneventful, but some quirks have been encountered in the radio protocols. It seems that when the start time of the relay sessions do not closely match between PHX and MRO, there is an issue where PHX may not be prepared to return data to the MRO radio, and some frame accounting goes badly. I'll let the radio experts try to explain it in better English later.

From last night, the two other unreported overflights looked like the following:
-- 183_06 - Nominal overflight, with a small roll towards PHX, with a small amount of data returned. All went as expected. PHX did not model the roll, but MRO behaved appropriately.
-- 183_07 - Nominal overflight, with a large roll towards PHX, with radiometric data collection included with a large data volume. All went as expected, and a radiometric data product was produced (which PHX has not looked at). PHX did not model the roll, but MRO behaved appropriately.

Overflights from this morning looked like the following:
-- 183_11 - Nominal overflight. PHX did not reply; this was fine.
-- 183_12 - Nominal overflight. Last night, PHX made a "tactical" change to bump up the return-link data rate to 128. This link went badly due to the aforementioned issue.
-- 183_13 - Nominal overflight, with a small amount of data returned, as expected.

There is only one overflight scheduled for tonight:
-- 184_05 - Nominal overflight, with a small amount of data expected.

Due to the aforementioned issue, during the 1 pm tagup it was decided to continue the simulation into tomorrow to try to test out the startup issues with PHX. Therefore, we expect two more overflights to execute:
-- 184_11 - Nominal overflight, but PHX is not expected to respond.
-- 184_13 - Nominal overflight, with a large amount of data expected.

This afternoon, PHX did provide an updated Request APF (v6) that changed the start time and durations of 184_05 and 184_13 (which also had a rate change). These were processed, for the record, in 15 minutes between the delivery of the Request APF and the completion of the SCMF by the ASP. (Of course, on both of the past two days, I was waiting for it ... so I may be setting a bad precedent ...).

Tomorrow, the last Relay Tagup of the ORT is scheduled for 1 pm. The simulated portion of the test should conclude (really!) at 11 am Pacific tomorrow.

-Roy

So tomorrow is the last day of this training exercise, called a "Operational Readiness Test" or ORT. I can't wait for it to be done ... but then we get the wonderful opportunity to pull together "lessons learned", which will end up with action items, items to be verified, and an enormous amount of paperwork. All in the next months when we get to start this whole thing all over again! Busy, busy.