VSOM 7.8 recording sporadic

[jnietling]

Post some NTP changes and a VSOM upgrade from 7.6.x to 7.8.0, I'm experiencing sporadic recording and streaming video (small lab environment with 8 cameras, single video VLAN).  See screenshot of SASD with obvious gaps in recorded info, and every so often, I get the dreaded "Attempting to Reconnect. #1" message on all cameras at different times in SASD as well.  The recording failures and reconnect messages are not simultaneous for any two cameras, so I'm assuming this is not a network or server load issue.  Disk is nowhere near capacity.  Thoughts?

[Scott Olsen]

Interesting.  Does the IMS.LOG file give any clues as to the periodic connectivity problem?  At some 'layer', there does appear to be a connectivity issue.  It's able to establish a connection, but not maintain it.

Does this happen with all cameras?  Are the cameras perhaps restarting?  How are they powered?

[jnietling]

Cameras are being powered via PoE on a slightly older catalyst switch (3560G), and yes it's happening to all of them (again, at different times).  Where is IMS.LOG, BTW?  Thanks for the thoughts/suggestions.

[Scott Olsen]

What are the camera models?

Also, you can access the logs from the VSMC console, or... if you're comfortable at the linux CLI, /usr/BWhttpd/logs.

[jnietling]

Sorry for the long delay.  Our cameras are 6020s, 6030s, and 6400s.  I just checked the logs, and I'm seeing a couple of things that kind of stand out  (truncated output):

*********** Motion teardown: Forcing send of motion STOP event for window

*********** Sending motion START event for window [0]

...

...

<Device.cxx:9699> ] A stream was reported unreachable and was then recovered, making sure that non-streaming configs (
motion, contact closure, serial, analytics, etc) on the device are as expected.

None of my camera templates enable motion detection based recording.  They're all configured for continuous recording.  The second log entry that stands out is:

<Proxy.cxx:591> ] Streaming error. Device disconnected or network error.

<RtpClient.cxx:1009> ] Timeout (10 secs) waiting for data from encoder.

<Proxy.cxx:164> ] Streaming error. Device disconnected or network error.

<RtpClient.cxx:1009> ] Timeout (10 secs) waiting for data from encoder.

These appear to be the evidence of the recording gaps.  Why the network errors?  In our smallish lab with all surveillance devices on the same VLAN, I'm not sure.  A few months ago, all was well, and now it's not.  We do have a lot of tinkering going on in our lab by multiple parties (daily ISE server config changes, etc), so it's entirely possible one of my colleagues has broken something.....but the randomness still leaves me scratching my head.

[Scott Olsen]

Unfortunately, it does sound like something is impacting reliable communication between VSMS instance and your endpoints.  I can't really speculate on the cause much beyond that.  My recommendation would be to try and correlate events and check for issues at layer 2 and 3 in the network.

[JDomeneck]

Did you happen to figure out a solution to correct your issues? I have the same scenario happening at my facility. Mine is on a larger scale though. I have approx. 215 cameras and these streaming blips are happening across all of them. I have a TAC case open but, I figure that I'd ask you while they are looking through the logs.

[Scott Olsen]

How many VSMS instances are these 215 cameras spread across?  Are they all housed in the same logical area of the network architecture?