The architecture is exactly as you listed it, VMware running on UCS blades with 5k upstream then 7k upstream.
I have looked at the VM stats, no packet drops at the time of the issue. Our F5 shows that the 2 servers became unavailable and were not responding to the health check.
No packet drops on the VMs in question
No host mem swapping
No CPU contention on the host
small deviation in storage latency but the spike doesn't even touch 1ms (Thanks HDS there is no better array)
SMall deviation in network throughput but the spike is only to 400KBps, we have 10Gbps networking
My review of the VMware stack shows its not the culprit
UCSM shows 0 ports stats for all loss counters, and 0 for all errors.
The event logs don't go back far enough, they must be getting over written, can't recall how that is configured, but I can only see the past week or so of events and that's it. I navigated to the specific blades that VMs were running on to check for faults and events and there was no data showing in either. Not sure why the events section would be empty though, I figured there should be some events logged but nothing.
I simply want to understand where I can see if packet drops took place within the UCS stack, I know where to look at this from the VMware host perspective, but not the UCS hardware per se, or if its something that is being monitored and logged in the stack.
I figured that if an interface in the UCS stack went down I would see this in the event logs, but as I state, the logs aren't going back far enough for me to see if something happened. I wasn't pulled in to try to diagnose this issue till over a week after it occurred. With VMware I can get a granular look at the stats using the Vrealize operations manager tool, but I have nothing like this tool for Cisco UCS and could certainly take a recommendation on something that might capture events better for historical analysis.
