Problem with M1 linecard on Nexus7000

[Nguyen Duc Trung]

Hi all,

We have a problem with some interface on M1 linecard. When I use show module command, the output show this module is Fail as bellow: 

CORE-NX7010-002# sh module

Mod  Ports  Module-Type                         Model              Status

---  -----  ----------------------------------- ------------------ ----------

1    48     10/100/1000 Mbps Ethernet Module    N7K-M148GT-11      ok

2    32     10 Gbps Ethernet Module             N7K-M132XP-12      ok

5    0      Supervisor module-1X                N7K-SUP1           ha-standby

6    0      Supervisor module-1X                N7K-SUP1           active *

Mod  Online Diag Status

---  ------------------

1    Pass

2    Fail --> This is M1 linecard

5    Pass

6    Pass

When I show logging on switch, there are some output relate to this linecard: 

SB-CORE-NX7010-002# sh logging

2014 May 31 03:12:33.352 SB-CORE-NX7010-002 %DIAG_PORT_LB-2-PORTLOOPBACK_TEST_FAIL: Module:2 Test:PortLoopback failed 10 cons

ecutive times. Faulty module: affected ports:10,14,16  Error:Loopback test failed. Packets lost on the SUP in the transmit direction

2014 May 31 03:12:33.360 SB-CORE-NX7010-002 %MODULE-4-MOD_WARNING: Module 2 (Serial number: JAF1303EBPK) reported warning on

ports 2/10-2/10 (Ethernet) due to Loopback test failed. Packets lost on the SUP in the transmit direction in device DEV_UNDEF

 (device error 0x41830059)

2014 May 31 03:12:33.360 SB-CORE-NX7010-002 %MODULE-4-MOD_WARNING: Module 2 (Serial number: JAF1303EBPK) reported warning on

ports 2/14-2/14 (Ethernet) due to Loopback test failed. Packets lost on the SUP in the transmit direction in device DEV_UNDEF

 (device error 0x41830059)

Can someone explain for me what is problem. Any suggestion for this case ?

Thanks,

Trung. 

[Reza Sharifi]

Hi,

From the logs output, it appears you have a bad module.  If you have a support contract with Cisco, open a Ticket with TAC and replace the module.

HTH

[Nguyen Duc Trung]

Hi Zeza,

Can you explain some root cause of this problem ?

Thanks,

Trung.

[Bilal Nawaz]

Hello,

I know of Bug(s) which are very similar to this one on the N7K's. CSCua03125, though it seemed to only affect the F2 line cards. Also CSCuf03533.

You will be able to get some further explanation from Cisco TAC. If you have a valid support contract there should not be a problem replacing it if it is faulty. It may have been a DOA (if its a new card) - I had this with an M1 card before.

Hth.

[Nguyen Duc Trung]

Hi Bilal,

So with this error, you have replaced linecard with supportted from TAC Team ? 

Thanks,

Trung.

[Bilal Nawaz]

Hello Trung,

Yes I had similar problem, line card error'd because it was DoA (Dead on Arrival) which means the card was shipped, and arrived non working. So the Cisco Reseller who support us RMA's back to Cisco and they sent new one within 24hrs.

But this was on N7009 chassis not 10. Have you seen the bug ID's? First contact TAC, they will work out if the line card is dead or if it is a bug.

[Nguyen Duc Trung]

Hello Bilal,

Thank you very much for your response. I has requested for TAC Support. Any information, I will inform. 

Thanks,

Trung.

[dtran]

Hi Trung, how are you ??

I've a pair Nexus 7010's with N7K-M132XP-12L cards and I've ran into the exact same problem. Looked like Cisco had a bad batch of M1 cards shipped out to customers. Unfortunately in my case I've had four incidents where the card fails so I've gone through four replacements with TAC. The latest incident was last Sunday where the whole line card was dead. The problem is ASIC related. I've got my Cisco account team involved now to further investigate. I can't believe I've had four replacements over the period of 12 months. So you not the only one out there. Please keep me updated Trung !!

Thanks !!

Danny

Shoot me an email if you would like to discuss further.

[menzies456]

Yes we have had 2 fail with the same customer in the past few weeks.

M7K-M132XP-12L cards.

Error seen

%MODULE-2-MOD_SOMEPORTS_FAILED: Modu
le 1 (Serial number: JAF1310AJJA) reported failure on ports xx (Etherne
t) due to Naxos: Fatal interrupt in device DEV_NAXOS (device error 0xc7201211)

Thanks

Sam

[dtran]

Hi Sam, how's going ??

You are not the only one Sam. I've gone through 4 replacements so far in about a year time frame, my most recent one was back in May.

Could you share some info on the symptoms that you've seen when the card fails ?

Shot me an email if you want to discuss further.

Thanks !!

Danny

[menzies456]

Hi,

Yes we see some ports go into error disabled and create many crc's to the other end of the connection.

Errors seen above.

Our customer wants a TAC case before they swap out the line card as they can live without the failed ports (only 2 ports on the same asic) are affected .

Will let you know what Cisco say.

Sam

[dtran]

Thanks Sam !!

I've ran into the exact same problem. TAC will send a replacement for sure.

I've seen where the card just completely dead (no power), port error-disabled, port flapping. I have a pair of N7010's running vPC and all the failures that I've had is only occurring on one chassis and Cisco is not convince that there is any issue related to the chassis.

Are you having the issue on the same chassis ?

Danny

[menzies456]

2 different chassis have the issues.

one line card has already been swapped.

I'm expecting this one to be done soon also.

Thanks

Sam

[dtran]

Hi Sam,

May I ask what version of NX-OS are you running ?

Thanks !! Please keep me in touch !!

Danny

[menzies456]

HI,

Just had another strange one that sound so similar, 6.2(8a),

Some ports just go into err-disabled and the modules are failing online diagnostic checks.

It is a pair of N7K but this time the VPC is showing as inconsistent type 2, svi saying that there are some down on the local device but up on the standby.

Well the local device does not show them as down.

Also some loss of mgmt traffic across the pair.

1 module in N7K1 is showing fail and 2 modules in N7K2,

All are the M132xp-12 type.

Nothing seen in the logs yet.

Also a strange physical port error = udld empty, echo, port:error.

Cabling changed but no changes.

Anyone seen?

Thanks

Sam