Re: FIB-DFC4-3-NOMEM: Malloc Failure, disabling DCEF on linecard

tf2-conky · ‎04-05-2010

Hi all

Immediately after a reload of our 6509e switch, it appears that DCEF has disabled on one of the Gig module daughter card's because of a malloc failure due to memory fragmentation.

Apr 1 02:23:45.296 xxST: %BGP-5-ADJCHANGE: neighbor x.x.65.28 Up

Apr 1 02:23:46.388 xxST: %STANDBY-6-STATECHANGE: Vlanxxx7 Group 34 state Active -> Speak
Apr 1 02:23:48.164 xxST: %STANDBY-6-STATECHANGE: Vlanxx8 Group 16 state Active -> Speak
Apr 1 02:23:48.164 xxST: %STANDBY-6-STATECHANGE: Vlanxx8 Group 2 state Active -> Speak
Apr 1 02:23:48.628 xxST: %STANDBY-6-STATECHANGE: Vlanxx6 Group 14 state Active -> Speak
Apr 1 02:23:51.012 xxST: %BGP-5-ADJCHANGE: neighbor x.x.124.11 Down Peer closed the session
Apr 1 02:23:55.809 xxST: %BGP-5-ADJCHANGE: neighbor x.x.124.11 Up
Apr 1 02:24:46.160 xxST: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: no memory
Apr 1 02:24:47.928 xxST: %SYS-DFC4-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x203CC468, alignment 16
Pool: Processor Free: 76184 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "CEF LC IPC Background", ipl= 0, pid= 105
-Traceback= 203B3F28 203B6910 203CC470 20F3A054 20F19610 20F1DDCC 20F24B38 20F351DC 20F2D788 20F2DBB8 20F2DE94 20F2EB14
Apr 1 02:24:48.436 xxST: %FIB-DFC4-3-NOMEM: Malloc Failure, disabling DCEF on linecard
Apr 1 02:24:50.156 xxST: %BGP-5-ADJCHANGE: neighbor x.x.54.22 Up
Apr 1 02:25:07.030 xxST: %BGP-5-ADJCHANGE: neighbor x.x.123.45 Up
Apr 1 02:26:09.994 xxST: %BGP-5-ADJCHANGE: neighbor x.x.124.19 Up
Apr 1 02:29:47.382 xxST: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window message, LC to RP IPC is non-operational

The module & daughter card otherwise appears to be fine

sh module

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

...

...
4 24 CEF720 24 port 1000mb SFP WS-X6724-SFP

...

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------

...

4 Distributed Forwarding Card WS-F6700-DFC3BXL SAL102504DN 5.3 Ok

...

Is there any way I can try and re-enable CEF on the line card? Failing this I was going to reseat the module, and then replace the RAM?

Ganesh Hariharan · ‎04-05-2010

Hi all

Immediately
after a reload of our 6509e switch, it appears that DCEF has disabled
on one of the Gig module daughter card's because of a malloc failure
due to memory fragmentation.
Apr  1 02:23:45.296 xxST: %BGP-5-ADJCHANGE: neighbor x.x.65.28 Up
Apr  1 02:23:46.388 xxST: %STANDBY-6-STATECHANGE: Vlanxxx7 Group 34 state Active -> Speak
Apr  1 02:23:48.164 xxST: %STANDBY-6-STATECHANGE: Vlanxx8 Group 16 state Active -> Speak
Apr  1 02:23:48.164 xxST: %STANDBY-6-STATECHANGE: Vlanxx8 Group 2 state Active -> Speak
Apr  1 02:23:48.628 xxST: %STANDBY-6-STATECHANGE: Vlanxx6 Group 14 state Active -> Speak
Apr  1 02:23:51.012 xxST: %BGP-5-ADJCHANGE: neighbor x.x.124.11 Down Peer closed the session
Apr  1 02:23:55.809 xxST: %BGP-5-ADJCHANGE: neighbor x.x.124.11 Up
Apr  1 02:24:46.160 xxST: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: no memory
Apr  1 02:24:47.928 xxST: %SYS-DFC4-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x203CC468, alignment 16
Pool: Processor  Free: 76184  Cause: Memory fragmentation
Alternate Pool: None  Free: 0  Cause: No Alternate pool
-Process= "CEF LC IPC Background", ipl= 0, pid= 105
-Traceback= 203B3F28 203B6910 203CC470 20F3A054 20F19610 20F1DDCC 20F24B38 20F351DC 20F2D788 20F2DBB8 20F2DE94 20F2EB14
Apr  1 02:24:48.436 xxST: %FIB-DFC4-3-NOMEM: Malloc Failure, disabling DCEF on linecard
Apr  1 02:24:50.156 xxST: %BGP-5-ADJCHANGE: neighbor x.x.54.22 Up
Apr  1 02:25:07.030 xxST: %BGP-5-ADJCHANGE: neighbor x.x.123.45 Up
Apr  1 02:26:09.994 xxST: %BGP-5-ADJCHANGE: neighbor x.x.124.19 Up
Apr  1 02:29:47.382 xxST: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window message, LC to RP IPC is non-operational
The module & daughter card otherwise appears to be fine
sh module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
...
...
  4   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP
...
...
...
...
Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
...
  4  Distributed Forwarding Card WS-F6700-DFC3BXL   SAL102504DN  5.3    Ok
...
...
...
...
...
Is
there any way I can try and re-enable CEF on the line card?  Failing
this I was going to reseat the module, and then replace the RAM?

Hi,

Following is the error explanation and recommended action as per cisco output interperter:-

%FIB-3-FIBDISABLE (x1): Fatal error, slot [dec]: [chars]

Explanation: An internal software error has occurred because of an IPC problembetween the LC and the RP. CEF switching has been disabled on the specified slot.

Recommended Action: Enter the clear cef linecard slot command. Next, check the status of the line card by entering the show cef linecard command. If the status shows that the line card is still disabled, disable and reenable dCEF and check the status of the line card again.

Check out the below link for command to do the same :-

http://www.ict-partner.net/en/US/docs/ios/12_1/switch/configuration/guide/xcdcefc.pdf

Hope to Help !!

Ganesh.H

Remember to rate helpful post

tf2-conky · ‎04-05-2010

Thanks for that.

I take it there is no way just to enable DCEF only on the line card?

Will re-enabling DCEF globally flush and rebuild the tables for the entire switch, or just the line card that DCEF is currently not runnning on?

It would be good to know what sort of impact this command might have etc.

Ganesh Hariharan · ‎04-06-2010

Thanks for that.

I take it there is no way just to enable DCEF only on the line card?

Will re-enabling DCEF globally flush and rebuild the tables for the entire switch, or just the line card that DCEF is currently not runnning on?

It would be good to know what sort of impact this command might have etc.

Hi,

As per the cisco it says to enable or disable the dcef on line card and generally there are no adverse side effects from enabling CEF.The main one to watch out for is that certain debugging will not work for packets that are fast switched CEF is a "fast" switching method that allows the processing effort
required to "switch" a packet to be substantially reduced when compared to the traditional method of simply looking up the routing table for each packet.

It has the advantage over the other methods of NOT needing to process switch the first packet to each destination and can also I suspect do per destination and per packet load balancing

Check out the below link for more information

http://www.cisco.com/en/US/tech/tk827/tk831/technologies_white_paper09186a00800a62d9.shtml

Hope to Help !!

Ganesh.H

Remember to rate the helpful post

FIB-DFC4-3-NOMEM: Malloc Failure, disabling DCEF on linecard (Cat 6509e)