Cisco 64Gb SPF and HPE Virtual Connect SE 32Gb connection issues

[twigs]

Hi!

I have and HP Synergy blade chassis with HPE Virtual Connect SE 32Gb with MPO to 4x LC Breakout Multi-Mode Fiber Optic Cable.

During testing of different hardware prior to going into production with new MDS switches, I found problem with this combination.

I have connected the HP VC and configured only two of four MPO uplinks as 16Gb, connected 2 Optic Cables to 2 Cisco 64Gb SFP, the ports are in up state and negotiate as 16Gb, If I change in the HP VC the configuration of the ports to 32Gb, I can see in the MDS switch port status as notConnected.

What could be the cause of this behavior?

Thanks

[marce1000]

  - Check logs on the MDS after trying to  make an intended connection (or configuration) ,

  M.

[twigs]

Thanks Marce, a colleague is working with HP support to analyze the cause of the problem.

I will report the cause when we have an answer.

[Jason Mooney · Accepted solution]

HPE is aware of this issue.  The issue comes down to the 100Gb QSFP to quad 32Gb SFPs have a problem.  While negotiating 32Gb FC speeds, the HPE SFP suddenly reduces it's speed to 16Gb when communicating back to the Cisco 64Gb interfaces.  This is caused by the HPE hardware expecting a Training Complete before it successfully finishes the negotiation.

The MDS isn't seeing any primatives or FC data on the line. You can use slot X show hardware internal fcmac port Y port-logs for interface fcX/Y and see that SNR:0.0 which indicates the MDS isn't receiving signal.  In this particular case we can see the fibre channel training frames agreed on 32Gb, but then the remote side droped to 16Gb.

`slot 3 show hardware internal fcmac port 1 port-logs`
========= FCP Port#1 Internal Port platform Log ==========
MMDDYY HHMMSS usecs   Event                  DATA
--------------------  -----                  -------------
041525 144012 314194  SERDES_RX_STATE        ST:0x4FF2207 DP:0x1 S:1 C:1 P:0 LEQ:0 VGA:25 ATT:2 SNR:0.0 FBF:0.00 1007:0x5D6F 1006:0xFE004804
041525 144012 313839  TTS_SYNC               Pass:1 val:0x104 RX:32G TX:16G TF.CS 0x L[08,70] R[C0,40]
041525 144012 282959  RX_LANE_ENABLE         Lane:0x31 SN:0x835 TX:0xC RX:0x10
041525 144012 282696  SERDES_DELAY           Reg val:4
041525 144012 282271  SERDES_RX_SPEED        32G 
041525 144012 282271  SERDES_TTS_RX          32G SN:0x835 Lane:0x11

This is an example of the Signal to Noise Ration being zero.

========= FCP Port#1 Internal Port platform Log ==========
MMDDYY HHMMSS usecs   Event              DATA
--------------------  -----              -------------
041525 144012 609444  SERDES_RX_DFE      0.00,0.00,0.00,2.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
041525 144012 609190  SERDES_RX_STATE    ST:0x4FF2207 DP:0x1 S:1 C:1 P:0 LEQ:0 VGA:25 ATT:2 SNR:0.0 FBF:0.00 1007:0x5D6F 1006:0xFE006835
041525 144012 608827  TTS_SYNC           Pass:1 val:0x104 RX:32G TX:32G TF.CS 0x L[C8,70] R[C0,40]
041525 144012 578502  SERDES_RX_DFE      0.00,0.00,0.00,2.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
041525 144012 578178  SERDES_RX_STATE    ST:0x4FF2207 DP:0x1 S:1 C:1 P:0 LEQ:0 VGA:25 ATT:2 SNR:0.0 FBF:0.00 1007:0x5D6F 1006:0xFE00681B
041525 144012 577809  TTS_SYNC           Pass:1 val:0x104 RX:32G TX:32G TF.CS 0x L[C8,70] R[C0,40]
041525 144012 547505  SERDES_RX_DFE      0.00,0.00,0.00,2.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
041525 144012 547181  SERDES_RX_STATE    ST:0x4FF2207 DP:0x1 S:1 C:1 P:0 LEQ:0 VGA:25 ATT:2 SNR:0.0 FBF:0.00 1007:0x5D6F 1006:0xFE006802
041525 144012 546810  TTS_SYNC           Pass:1 val:0x104 RX:32G TX:32G TF.CS 0x L[C8,70] R[C0,40]

The recommendation is to Open a case with HPE and ask them for assistance.  You can also open a case with Cisco TAC to request assistance.