PHD 4x camera upgraded unexpectedly (SX20 codec)

[mikebuscema]

Hi,

I support Cisco SX20 codecs that are integrated into telehealth carts.  A customer reported an issue where after a reboot, the camera started upgrading.  I don't know when the last reboot occurred, but there was no software upgrade done on the codec, nor was the camera swapped out since. The camera upgrade takes a while, so they were unable to use the cart for patient care during that time and I've been tasked with finding root cause for this event.

The codec shipped with CE8.3.5, and the cart ships with the camera detached.  It's been in use since September or October of 2018 and has not been upgraded or otherwise altered.  The only time a camera should be upgraded is after a codec upgrade, or if it's been newly connected to a codec with newer software.  Any ideas?

[Wayne DeNardi]

Assuming it is still the same camera, and the codec has not had a software update at all (perhaps it hadn't been used in a while during that period of time when a software update was pushed?) - then the only explanation I can think of is perhaps the camera control cable wasn't 100% connected properly before, but connected enough to power the camera (and the video goes over the HDMI), then, as the cable has connected properly, the codec has seen the camera has an old software release on it, and it updated it (as if it had just been plugged in).

As per the release notes, the last camera software update for the included in CE8.x for the 4x camera was in CE8.3.4.

[mikebuscema]

Thanks for the response, Wayne.  So with that in mind, I assume it's not possible to truly know the root cause even by looking through the codec logs?

[Wayne DeNardi]

You may be able to find something by trawling through the log files, but I'd say that anything you find won't be anywhere near as descriptive as you'd want it to be - you would just get something along the lines of "it's on an old software version, so I'm going to update it".  I haven't got anything that has done a software update of a camera here for a long time, so I don't have anything I can check for you.

[mikebuscema]

Thanks again Wayne...so in looking through the logs I did see that it looks like the codec "upgraded" to CE8.3.5 on 6/12, which corresponds to around the time the issue was reported.  See the info from the messages.log file below:

2019-06-12T17:30:00.000-05:00 (none) swupgrade: UDI: CTS-SX20N-CODEC FTT2149008Y
2019-06-12T17:30:01.000-05:00 (none) swupgrade: Serial number: FTT2149008Y
2019-06-12T17:30:01.000-05:00 (none) swupgrade: ce8.3.5.c877c7f
2019-06-12T17:30:01.000-05:00 (none) swupgrade: SW Release Date: 2018-06-07

I can certainly bring this up to the customer, but he already assured me they did not push an upgrade to the codec, and regardless, the codec was already on 8.3.5 when we shipped it in September of 2018 so the camera should have upgraded when it was first connected.

Sorry for pressing, but I know that providing this data to the customer is going to raise more questions...

[Wayne DeNardi]

That certainly looks like an upgrade was pushed to the codec.  Are you able to see back any further in the logs to see precisely what version the codec had on it before (you believe it had the right version, but the log file will confirm that)?

[mikebuscema]

Looking at every messages.log file in both the current and historical logs, this is what I've found. Each instance (4 lines) is from a different log file:

2018-06-06T19:00:34.000-05:00 SEP247E12F05F67 swupgrade: UDI: CTS-SX20N-CODEC FTT2149008Y
2018-06-06T19:00:34.000-05:00 SEP247E12F05F67 swupgrade: Serial number: FTT2149008Y
2018-06-06T19:00:34.000-05:00 SEP247E12F05F67 swupgrade: ce8.3.5.c877c7f
2018-06-06T19:00:34.000-05:00 SEP247E12F05F67 swupgrade: SW Release Date: 2018-06-07

2018-12-28T16:26:36.000-06:00 (none) swupgrade: UDI: CTS-SX20N-CODEC FTT2149008Y
2018-12-28T16:26:36.000-06:00 (none) swupgrade: Serial number: FTT2149008Y
2018-12-28T16:26:36.000-06:00 (none) swupgrade: ce8.3.5.c877c7f
2018-12-28T16:26:36.000-06:00 (none) swupgrade: SW Release Date: 2018-06-07

2019-01-02T15:49:12.000-06:00 (none) swupgrade: UDI: CTS-SX20N-CODEC FTT2149008Y
2019-01-02T15:49:12.000-06:00 (none) swupgrade: Serial number: FTT2149008Y
2019-01-02T15:49:12.000-06:00 (none) swupgrade: ce8.3.5.c877c7f
2019-01-02T15:49:12.000-06:00 (none) swupgrade: SW Release Date: 2018-06-07

2019-02-27T05:20:49.000-06:00 (none) swupgrade: UDI: CTS-SX20N-CODEC FTT2149008Y
2019-02-27T05:20:49.000-06:00 (none) swupgrade: Serial number: FTT2149008Y
2019-02-27T05:20:49.000-06:00 (none) swupgrade: ce8.3.5.c877c7f
2019-02-27T05:20:49.000-06:00 (none) swupgrade: SW Release Date: 2018-06-07

2019-06-12T17:30:00.000-05:00 (none) swupgrade: UDI: CTS-SX20N-CODEC FTT2149008Y
2019-06-12T17:30:01.000-05:00 (none) swupgrade: Serial number: FTT2149008Y
2019-06-12T17:30:01.000-05:00 (none) swupgrade: ce8.3.5.c877c7f
2019-06-12T17:30:01.000-05:00 (none) swupgrade: SW Release Date: 2018-06-07

So it looks like the codec has been pushed the CE8.3.5 software 5 times since June of 2018, yet the camera didn't upgrade until June 12, 2019.  Does that look right?  And is there any way to tell how/why the upgrade is being pushed (i.e. by CUCM, TMS, etc.)?

[mikebuscema]

I just checked the current messages.log file from one of our SX20s and it shows the same "upgrade" occurred on 8/22 (two weeks ago). I'm certain there was no upgrade pushed, either manually by one of my team or automatically by CUCM.  It also already had CE8.3.5 installed prior to this message:

2019-08-22T19:43:41.000+00:00 (none) swupgrade: UDI: CTS-SX20CODEC FTT200500FC
2019-08-22T19:43:41.000+00:00 (none) swupgrade: Serial number: FTT200500FC
2019-08-22T19:43:41.000+00:00 (none) swupgrade: ce8.3.5.c877c7f
2019-08-22T19:43:41.000+00:00 (none) swupgrade: SW Release Date: 2018-06-07

Does anyone know why CE8.3.5 would be pushed to the SX20 repeatedly when the codec already has that version installed?