Re: cisco 4507 R

sharma16031981 · ‎04-11-2011

Hi Friends,

last saturday I faced one problem in our core switch 4507 R . Active sup lost connection and standby came active. We got lot of errors/alerts on console shown below.

%SYS-2-MALLOCFAIL: Memory allocation of 784 bytes failed from 0x101ED660, alignm
ent 8
Pool: Processor Free: 50856 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 6
-Traceback= 101BC86C 10262B94 10267F20 101ED664 101F1C7C 101F1DA0 1029E52C 10294
    170 permit ip 10.2.150.0 0.0.0.255 10.2.14.0 0.0.0.255 (6918 matches)
    180 permit ip 10.2.14.0 0.0.0.255 10.2.151.0 0.0.0.255 (1554536 matches)
    190 permit ip 10.2.151.0 0.0.0.255 10
    200 permit ip 10.2.14.0 0.0.0.255 10.2.1.0 0.0.0.255 (40996 matches)
    210 permit ip 10.2.1.0 0.0.0.255 10.2.14.0 0.0.0.255 (45813 matches)
    220 deny ip 10.2.14.0 0.0.0.255 10.2.0.0 0.0.255.255 (8104 matches)
    230 deny ip 10.2.0.0 0.0.255.255 10.2.14.0 0.0.0.255 (40 matches)
--More--
       %IPC-4-GET_PAK_MSG: Failed for message size=0acket
-Traceback= 101BC86C 1056E53C 10568754 1058094C 1056E904 1056EC98 1057F7B0 1056F
2F0 1059CBE4 105714A4 1057F584 1029E52C     240 deny ip 10.2.14.0 0.0.0.255 10.
2.0.0 0.0.255.255
    250 deny ip 10.22.0.0 0.0.255.255 10.2.14.0 0.0.0.255
    260 deny ip 10.2.14.0 0.0.0.255 10.21.0.0 0.0.255.255
    270 deny ip 10.21.0.0 0.0.255.255 10.2.14.0 0.0.0.255
    280 deny ip 10.2.14.0 0.0.0.255 10.31.0.0 0.0.255.255
    290 deny ip 10.31.0.0 0.0.255.255 10.2.14.0 0.0.0.255
    300 deny ip 10.2.14.0 0.0.0.255 10.32.0.0 0.0.255.255
    310 de
%CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet6
/38 (1), with rmsisw14 FastEthernet0/48 (30).
%% Low on memory; try again lat
%SYS-2-MALLOCFAIL: Memory allocation of 784 bytes failed from 0x101ED660, alignm
ent 8
Pool: Processor Free: 50920 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Spanning Tree", ipl= 0, pid= 52
-Traceback= 101BC86C 10262B94 10267F20 101ED664 101EDBD0 101F0380 101ECB4C 10434
6CC 10434CD8 104377AC 10439330 1041A38C 1041A574 1029E52C 10294EC8er

%% Low on memory; try again later

%% Low on memory; try

%% Low on memory; try again later

%DUAL-3-NOBUFFER: No buffers available for 60 byte packet
%% Low on memory; try again later

%% Low on memory; try again later

%IPC-4-GET_PAK_MSG: Failed for message size=0
-Traceback= 101BC86C 1056E53C 10568754 1058094C 1056DBEC 1057F4D4 1029E52C 10294
EC8
%% Low on memory; try again later

%% Low on memory; try again later

%SYS-2-MALLOCFAIL: Memory allocation of 784 bytes failed from 0x101ED660, alignm
ent 8
Pool: Processor Free: 50216 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 6
-Traceback= 101BC86C 10262B94 10267F20 101ED664 101F1C7C 101F1DA0 1029E52C 10294
EC8
%% Low on memory; try again later

%% Low on memory; try again later

%% Low on memory;

%% Low on memory; try again later

%DUAL-3-NOBUFFER: No buffers available for 60 byte packet
%IPC-4-GET_PAK_MSG: Failed for message size=0
-Traceback= 101BC86C 1056E53C 10568754 1058094C 1056DBEC 1057F4D4 1029E52C 10294
EC8
%% Low on memory; try again later

%% Low on memory; try again later

%CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on GigabitEthernet6
/38 (1), with rmsisw14 FastEthernet0/48 (30).
%% Low on memory; try again later

%% Low on memory; try again later

%SYS-2-MALLOCFAIL: Memory allocation of 784 bytes failed from 0x101ED660, alignm
ent 8
Pool: Processor Free: 50472 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "IP Input", ipl= 0,
-Traceback= 101BC86C 10262B94 10267F20 101ED664 101EDBD0 101F0380 101ECB4C 10750
A58 10751660 10751168 107511F0 1074791C 10015198 101ECE14 1073EEF8 10730E84
%% Low on memory; try again later

%% Low on memory; try again later

%DUAL-3-NOBUFFER: No buffers available for 60 byte packet
%IPC-4-GET_PAK_MSG: Failed for message size=0
-Traceback= 101BC86C 1056E53C 10568754 1058094C 1056DBEC 1057F4D4 1029E52C 10294
EC8
%SYS-2-MALLOCFAIL: Memory allocation of 784 bytes failed from 0x101ED660, alignm
ent 8
Pool: Processor Free: 50208 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "IP Input", ipl= 0, pid= 59
-Traceback= 101BC86C 10262B94 10267F20 101ED664 101EDBD0 101F0380 101ECB4C 10743
E68 10745B8C 1072FEDC 10730364 10730664 1029E52C 10294EC8
000900: .Apr 9 05:10:56.058 IST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.
2.12.6 (GigabitEthernet4/1) is up: new adjacency
000901: .Apr 9 05:10:59.494 IST: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 10.
2.12.2 (GigabitEthernet3/5) is up: new adjacency

After that I ran reload command and both sup got rebooted and they came to original state. 1 active and 2 standby.

againg we are getting some errors on console shown below.

001403: Apr 11 12:33:41.989 IST: %EM-4-SEARCH: The EM function (em_unlock_intern
al) searched for the specified open event (562),
but the search failed.
001404: Apr 11 12:33:41.989 IST: -Traceback= 105AFCD0 105AFBCC 105C83A8 105CD9F0
1044CA6C 1029E52C 10294EC8
001405: Apr 11 12:33:41.997 IST: %EM-4-SEARCH: The EM function (em_unlock_intern
al) searched for the specified open event (1227),
but the search failed.
001406: Apr 11 12:33:41.997 IST: -Traceback= 105AFCD0 105AFBCC 105C83A8 105CD9F0
1044CA6C 1029E52C 10294EC8
001407: Apr 11 12:33:42.001 IST: %EM-4-SEARCH: The EM function (em_unlock_intern
al) searched for the specified open event (1238),
but the search failed.
001408: Apr 11 12:33:42.001 IST: -Traceback= 105AFCD0 105AFBCC 105C83A8 105CD9F0
1044CA6C 1029E52C 10294EC8

SHOW VERSION IS SHOW BELOW

--------------------------------------------------------------------------------------------------------------------------------------

Cisco IOS Software, Catalyst 4000 L3 Switch Software (cat4000-I9S-M), Version 12
.2(25)EWA9, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Wed 21-Mar-07 11:35 by tinhuang
Image text-base: 0x10000000, data-base: 0x114B64F4

ROM: 12.2(20r)EW1
Dagobah Revision 95, Swamp Revision 28

A7-Central uptime is 14 weeks, 1 day, 19 hours, 12 minutes
Uptime for this control processor is 2 days, 22 hours, 58 minutes
System returned to ROM by Stateful Switchover
System restarted at 15:45:03 IST Fri Apr 8 2011
System image file is "bootflash:cat4000-i9s-mz.122-25.EWA9.bin"

cisco WS-C4507R (MPC8245) processor (revision 7) with 524288K bytes of memory.
Processor board ID FOX07360145
MPC8245 CPU at 333Mhz, Supervisor IV
Last reset from Redundancy Reset
15 Virtual Ethernet interfaces
112 Gigabit Ethernet interfaces
403K bytes of non-volatile configuration memory.

Configuration register is 0x2102

So, If someone can put some light on this issue and suggest what to do to resolve this. Also when I reloaded the switch with reload command only both sups got reloaded but I want to reload all the modules but reload command donot gives any options for that.

Thanks

Reza Sharifi · ‎04-11-2011

Hi Sharma,

When you reload the Sups, that will reload the entire switch including all your modules.

Can you provide the output of "sh red"?

Are your Sups exactly the same as far as hardware and software goes?

Leo Laohoo · ‎04-11-2011

%% Low on memory; try again later

It's always a bug.

12.2(25)EWA9, RELEASE SOFTWARE (fc3)

Please upgrade your IOS. I haven't seen this image since 2007!