Nightmare connecting N9K-C9396PX with M4PC-CFP2 to a N9K-C9364C

[ss1]

Hello,

I wonder if anybody ever used N9K-C9396PX with M4PC-CFP2 uplink board together with a CFP2 to QSFP28 converter. I have absolute nightmares getting it to link to a 9364C switch....

The CFP2 to QSFP28 converter is used on the M4PC-CFP2 in order to link through 2pcs. QSFP28 modules on both sides rather than to use CFP2 modules which are quite expensive nowadays. On the other side we have C9364C and a QSFP28 module on it. I had to do 'fec off' on the 9364C side in order to get the link up at all. It links and starts passing traffic through, however only when QSFP28-100G-SR4 with MPO/12 connector and patch cords are used. In case of the same port configuration and QSFP28-100G-LR4 with LC/UPC Duplex connectors, the link never goes up on the 9364C side however the 9396PX shows the port as flapping. 

In case of QSFP28-100G-SR4 connection I have unreliable transmission. Rx/Tx power levels are good on each side however there are CRC packet errors received on the 9364C through the converter module. Also the port would flap once daily without any clue logged in on any of the switches' syslogs. 

I have tried to force 100G with duplex full and no negotiate auto on each side however it couldn't help. Also the 9396PX doesn't support the 'fec off' command in order to disable it - it shows that the platform doesn't support FEC but anyway I wanted to have same configuration as much as possible. The NX-OS versions are 9.3.7 and the EPLD version is also 9.3.7. 

I'm not sure how could it be but the weirdest thing is that I just replace the SR4 modules with LR4 and the link won't connect anymore. I have also searched for some power setting in order to enable/disable power management feature but can't find. Is there any chance to have insufficient power delivered by the M4PC to the CFP2 converter? I'm stuck. Any ideas would be appreciated.

Thank you.

[ss1]

Happy new year to everyone. 

Let me confirm, the source of this issue has been isolated.

Cheapest CFP2 to QSFP28 do not have CDR function (Clock data recovery) which as far as I understand works to re-align the electrical signals, I mean, something like a FEC algorithm in the electrical layer or so. Sorry I may not have explained it very well because I'm not familiar enough with electrical circuit stuff but the important thing is that cheapest CFP2 to QSFP28 converters don't have such CDR circuit and the communication is unstable on electrical layer. 

I have given up looking for QSFP28 converters because those which are listed with CDR in the technical datasheets are EUR 1000 or so. I have resolved my case by purchasing some second hand pieces CFP2-LR4-100G modules and deployed LR4 connectivity between regular CFP2 modules and QSFP28 on the other side of the link. I can confirm both solutions work good: CFP2-LR4-100G towards a QSFP28 on the other switch, OR CFP2-LR4-100G towards another CFP2-LR4-100G in case a M4PC-CFP2 links to another M4PC-CFP2 on the other side. I have some dark fiber routes 15km or so working this way for months, no troubles ever. 

No solution for SR4 links yet. Unfortunately I can't go with expensive converters in order to plug SR4 in them because a 9396 switch + M4PC-CFP2 + 4 pcs.  converters cost twice as much as a 92160YC on third party dealers. I've prioritized the 92160, 93180 and 9364C in favor of 9396+M4PC-CFP2 where possible, they all cost more or less the same with all the peripherals summed up in the switch price. 

Apart from that M4PC-CFP2 are good boards. I can't say that I'm not happy with this purchase as they do their job over long haul links but for in-rack deployments I would rather go with QSFP28-equipped switches. 

[QuanNT86]

Thanks a lot!

Thanks.

[ss1]

Dear QuanNT86,

I may not be able to publish purchase advisories for non-genuine products or second hand products from unknown dealers to an official Cisco forum. Please take into account that we also discussed the issue on private messages. I would like to apologize for causing you inconvenience and would like to thank you for your thorough understanding.