Cisco C4900M, HP DL380-G9 and VMware 6.5

[TomasS]

Hi!

Having a very strange behaviour between six HP DL380p G9 servers and a Cisco C4900M switch

connected with DAC-cables (Cisco H10GB-CU1M), but also tested connection with a fibre cable with Cisco-coded SFP:s (SFP+) and got same result.

Problem: The C4900M reporting “err-disable” on some of the ports (not all) and a high packet loss (for testing ping between all servers).  The “err-disable” cause is “Link flaps”. Some of the servers is always giving “err-dissable”, some of them doing sometimes and some of them is never giving that error. Vmkernal log reporting the nic-card is going up an down.

Troubleshooting: The following tests is done, followed by the result.

1. Checked that all setup, config and versions of firmware, software is the same for the six HP servers and the six ports on the switch.
2. Connecting two servers directly to each other through DAC-cable. Result: no problems at all (0% packet loss), and ports is up all time.
3. Cross connect two ports on switch with DAC-cable (tested several cables and ports). Result: no “Link flaps” reported. Haven´t tested to ping.
4. Connected servers to switch with DAC-cable. Result: Some (of the six) ports on switch is reporting “Link flaps”, between 5-60% packet loss during ping from different servers and from switch.
5. Moved cable from one connection that not reporting “Link flaps” to one that is reporting. Result: No change (same server still reporting “Link flaps”).
6. Switched port on switch between one server that is reporting “Link flaps” and one that not is doing. Result: Same server still reporting “Link flaps”.
7. Switched CVR-X2-SFP10G adaptors (X2->SFP+) between to switch port that is reporting and not reporting “Link Flaps”. Result: No change.

Switch: A standalone C4900M with no other connections than the six servers (ok, management port is connected to a LAN).

  IOS-version: c4500e-entservicesk9-122.53.SG3. Ports settings is:

  Port config:  switchport mode access, switchport access vlan 1

Servers: All new HP DL380p G9 with ESXI 6.5 installed. Using one of the two 10 GB SFP+ ports of the “built in” 560FLR-card.

HP Product ID

852432-B21

  HP System ROM: P89 v2.56 (01/22/2018)

  HPE Ethernet 10GB 2-port 560FLR-SFP+Adapter  (driver: ixgbe; version: 4.5.2-iov; firmware-version: 0x80000838, 1.1752.0)

  VMware ESXi 6.5.0 build-7967591, VMware ESXi 6.5.0 Update 1

Have anyone an idea of root cause for the issue?