DFC Error on our Core Switch

[haidar_alm]

Hello,

I was looking this morning on some of the logs on our core routers and I found the following on one of them:

May 29 15:51:54 BST: %EARL-DFC3-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x46] JA_DR_RI_0_STA_FCO = 0x3

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x85] JA_DR_RI_1_STA_FCO = 0x0

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x28A] JA_FI_FT_RCV_RATE_SEL = 0xC

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x289] JA_FI_FT_XMIT_SHAPE = 0xFFF

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x4CB] JA_AG_PM_RAM_FULLNESS_0 = 0x0

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x4CC] JA_AG_PM_RAM_FULLNESS_1 = 0x20

May 29 15:51:54 BST: %HSRP-5-STATECHANGE: Vlan321 Grp 21 state Active -> Speak

May 29 15:51:54 BST: %HSRP-5-STATECHANGE: Vlan373 Grp 73 state Standby -> Active

May 29 15:51:54 BST: %HSRP-5-STATECHANGE: Vlan373 Grp 73 state Active -> Speak

May 29 15:51:54 BST: %HSRP-5-STATECHANGE: Vlan381 Grp 81 state Active -> Speak

May 29 15:51:54 BST: %HSRP-5-STATECHANGE: Vlan349 Grp 49 state Standby -> Active

May 29 15:51:54 BST: %HSRP-5-STATECHANGE: Vlan349 Grp 49 state Active -> Speak

May 29 15:51:55 BST: %HSRP-5-STATECHANGE: Vlan310 Grp 10 state Speak -> Standby

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x4CE] JA_AG_PM_RAM_FULLNESS_3 = 0x102

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x4F1] JA_AG_RM_RAM_FULLNESS_0 = 0x0

May 29 15:51:54 BST: %PF_ASIC-DFC3-3-ASIC_DUMP: [0:0x4F2] JA_AG_RM_RAM_FULLNESS_1 = 0x0

May 29 15:51:55 BST: %HSRP-5-STATECHANGE: Vlan328 Grp 28 state Standby -> Active

May 29 15:51:55 BST: %HSRP-5-STATECHANGE: Vlan328 Grp 28 state Active -> Speak

The logs goes on and on. As you can see I believe that this error at the top:

%EARL-DFC3-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0

Is the culpirt. Looking at the cisco site( http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsj72438) I found the following:

Control plane instability and %EARL-DFC3-2-SWITCH_BUS_IDLE: Switching bu

Symptom: Traffic stalling and switch bus idle messages: %EARL-DFC3-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0Conditions: most perceived when high utilization of the fabricWorkaround: None

Version of IOS we are running is:

12.2(33)SXH6

The Cisco Bug toolkit states that this was fixed in release SXH1, so surely SXH6 should've solved this, yet we are still getting this error!

Am I looking at a faulty hardware, or should a simple resear of the card solve this problem? Or are we even looking at an IOS upgrade on our core routers?

Any ideas please?

Many thanks,

[Ivan Shirshin]

Hello,

You may want to upgrade to 12.2(33)SXH08 or later - there is a DDTS CSCth55383 that causes same kind of alarms.

Otherwise it could be HW issue with either card or chassis slot - try with another card in this slot or same card in another slot to determine the point of failure.

Kind Regards,
Ivan

**Please grade this post if you find it useful.

[haidar_alm]

Many thanks for your reply.

Am I right in thinking that the issue is with the 3rd Module since the error message states:

%EARL-DFC3-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0

or

is it the 2nd Module that's got the problem because of the 2 in the error:

%EARL-DFC3-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0

When I run the follwoing command I get a pass on all the modules as stated below:

CORESWITCH#sh mod

Mod Ports Card Type                              Model              .

--- ----- -------------------------------------- ------------------ -----------

  1   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP     

  2   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP      

  3   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP    

  5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B      

  6    2  Supervisor Engine 720 (Hot)            WS-SUP720-3B      

  7    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE     

  8    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE     

Mod MAC addresses                       Hw    Fw           Sw           Status

--- ---------------------------------- ------ ------------ ------------ -------

  1  0019.aa50.8810 to 0019.aa50.8827   2.5   12.2(14r)S5  12.2(33)SXH6 Ok

  2  001a.6d73.4d70 to 001a.6d73.4d87   2.5   12.2(14r)S5  12.2(33)SXH6 Ok

  3  0019.aa70.0fd0 to 0019.aa70.0fe7   2.5   12.2(14r)S5  12.2(33)SXH6 Ok

  5  0017.9568.3ae8 to 0017.9568.3aeb   5.3   8.4(2)       12.2(33)SXH6 Ok

  6  0014.a982.10d8 to 0014.a982.10db   5.3   8.4(2)       12.2(33)SXH6 Ok

  7  0019.5500.5a2c to 0019.5500.5a2f   2.5   12.2(14r)S5  12.2(33)SXH6 Ok

  8  0019.2f56.bc14 to 0019.2f56.bc17   2.5   12.2(14r)S5  12.2(33)SXH6 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status

---- --------------------------- ------------------ ----------- ------- -------

  1  Distributed Forwarding Card WS-F6700-DFC3B     5  4.4    Ok

  2  Distributed Forwarding Card WS-F6700-DFC3B       4.5    Ok

  3  Distributed Forwarding Card WS-F6700-DFC3B       4.6    Ok

  5  Policy Feature Card 3       WS-F6K-PFC3B        2.3    Ok

  5  MSFC3 Daughterboard         WS-SUP720            2.6    Ok

  6  Policy Feature Card 3       WS-F6K-PFC3B         2.3    Ok

  6  MSFC3 Daughterboard         WS-SUP720            2.6    Ok

  7  Distributed Forwarding Card WS-F6700-DFC3B       4.6    Ok

  8  Distributed Forwarding Card WS-F6700-DFC3B       4.6    Ok

Mod  Online Diag Status

---- -------------------

  1  Pass

  2  Pass

  3  Pass

  5  Pass

  6  Pass

  7  Pass

  8  Pass

**************
Is there a way to find out which card is in question besides looking at the error log?

Many thanks,

[Ivan Shirshin · Accepted solution]

Hi,

The error is reported for EARL on module 3. You could check for this issue by the log messages or you could also check command "remote command module 3 show platform software earl reset history".

Kind Regards,
Ivan

**Please grade this post if you find it useful.

[ปลาวาฬทราย RMUTT CPE IX]

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsj72438