Hello,
Here it the description of the issue :
We have a system of 65 servers interconnected together thanks to CISCO switches WS-C3650-48TD.
Several servers send multicast traffic and one server is supposed to receive all multicast traffic. We observed some IP packet losses sometimes on this multicast traffic on servers receiving the traffic and our investigations concluded on a CISCO switch issue occurring when an important multicast traffic goes through one CISCO port. Elements that helped us reach this conclusion:
- We tried to subscribe other servers to the multicast traffic and we observe that IP packet loss is only on the server receiving the full multicast traffic. Other servers that subscribes only to part of the IP traffic do not have any IP packet losses => conclusion is that all IP multicast traffic is well received by the network switch.
- We inserted an IP analyzer (VTS500) on the link between the CISCO switch and the server receiving all traffic. We observed that traffic sent by network switch has some IP packets missing : This conclude on an issue on the CISCO network switch.
Note that we have a backup network switch and the behavior is the same so the issue seems to be a firmware issue. We also tested with another CISCO switch (2960) and the issue is the same (general issue on CISCO switches?).
We had a look to port counter stats & buffer stats(hit & missed) and no packet loss is detected by the switch when we observe packet losses. We tried also to change the output hold queue from 40 to 2000 on the switch but it does not solve the issue.
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 52 WS-C3650-48TD 03.03.05SE cat3k_caa-universalk9 INSTALL
It happens many times depending of the number of service present in the MPTS
4 services in mpts (20Mb/s) every 15-60 minutes
15 services in mpts (50Mb/s) every 2-15minutes
The usual bandwidth for this system is 80Mb/s
Sometimes you can have some burst out of the VS encoding, the following checks have been done :
- The cicsco statistics does not show any lost of packet. It should have shown it if we were in overflow issue.
