configur from L2 TO L3 and after 3 days I lose the connectivity between the core, Distribution

[coolj_212]

my old technology was working as l2 and we decided to change from L2 to L3 And we configure our ospf between Distribution to the core and after 3 days I lose the connectivity between the core, Distribution so I thought the issue is in the Distribution so I configure the Distribution as L2 and it was working Fine for more than one month last week I configure the Distribution as L3 and after 3 days I loss the connectivity so when I chick the Distribution I found the Distribution is not working As VSS

VSSDS-# show switch virtual role

xecuting the command on VSS member switch role = VSS Active, id = 1

RRP information for Instance 1

--------------------------------------------------------------------

Valid Flags   Peer     Preferred Reserved

               Count     Peer       Peer

--------------------------------------------------------------------

TRUE   V       0           0        0

Switch Switch Status Preempt       Priority Role     Local   Remote

Number         Oper(Conf)   Oper(Conf)         SID     SID

--------------------------------------------------------------------

LOCAL   1     UP     FALSE(N )     100(100) ACTIVE   0       0  

Peer 0 represents the local switch

Flags : V - Valid

In dual-active recovery mode: No

I in L2 and it was working for more

VSSDS-# show switch virtual role

xecuting the command on VSS member switch role = VSS Active, id = 1

Switch mode                 : Virtual Switch

Virtual switch domain number : 2

Local switch number         : 1

Local switch operational role: Virtual Switch Active

Cannot execute command on peer supervisor because it is not in SSO Standby state

VSSDS-# show switch virtual link

Executing the command on VSS member switch role = VSS Active, id = 1

VSL Status : DOWN

VSL Uptime : -

VSL Control Link : -

VSL Encryption : Configured Mode - Off, Operational Mode - Off

Cannot execute command on peer supervisor because it is not in SSO Standby state

The IOS for the 2 Distribution is : cat4500e-universalk9.SPA.03.05.00.E.152-1.E.bin

And also I chick the Debug I found

*Sep 26 04:28:04.428: %C4K_REDUNDANCY-5-CONFIGSYNC: The private-config has been successfully synchronized to the standby supervisor

*Sep 26 04:28:06.025: %C4K_REDUNDANCY-5-CONFIGSYNC: The startup-config has been successfully synchronized to the standby supervisor

*Sep 26 04:31:49.463: %C4K_REDUNDANCY-5-CONFIGSYNC: The private-config has been successfully synchronized to the standby supervisor

*Sep 26 04:31:50.894: %C4K_REDUNDANCY-5-CONFIGSYNC: The startup-config has been successfully synchronized to the standby supervisor

*Sep 27 21:42:17.243: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Spanning Tree", ipl= 0, pid= 201

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+16EFE7C :1F6A5000+16EB6CC :1F6A5000+3F97448 :1F6A5000+3F97BF8 :1F6A5000+3F61CC4

*Sep 27 21:42:47.314: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Spanning Tree", ipl= 0, pid= 201

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+16EFE7C :1F6A5000+16EB6CC :1F6A5000+3F97448 :1F6A5000+3F97BF8 :1F6A5000+3F61CC4

*Sep 27 21:43:17.384: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Spanning Tree", ipl= 0, pid= 201

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+16EFE7C :1F6A5000+16EB6CC :1F6A5000+3F97448 :1F6A5000+3F97BF8 :1F6A5000+3F61CC4

*Sep 27 21:43:47.398: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:44:17.406: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:44:47.459: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:45:17.475: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:45:47.487: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:46:17.519: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:46:47.559: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:47:17.583: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:47:47.595: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:48:17.599: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:48:47.612: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:49:17.619: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:49:47.647: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:50:17.703: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

-Traceback= 1#49176b00b95a50f3145e3825de17d470 :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+4182EC8 :1F6A5000+41830A8

*Sep 27 21:50:47.712: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

*Sep 27 21:51:17.747: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8

*Sep 27 21:51:47.778: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O Free: 9096 Cause: Memory fragmentation

Alternate Pool: None Free: 0 Cause: No Alternate pool

-Process= "Pool Manager", ipl= 0, pid= 8’

When I restart the Distribution every think work Fine can you Please Advise

[Georg Pauwen]

Hello,

post a schematic drawing of your topology. Which platforms are you using ? Also post the configs of both OSPF enabled neighbors...

[coolj_212]

interface Port-channel44

 description connected to DIS- port-channel 44

 no switchport

 ip address 10.x.x.x 255.255.255.252

 no ip redirects

 no ip unreachables

 no ip proxy-arp

interface TenGigabitEthernet2/3/3

 description To | SWITCH |

 no switchport

 no ip address

 no ip redirects

 no ip unreachables

 no ip proxy-arp

 channel-group 44 mode on

!

interface TenGigabitEthernet2/3/4

 description To | SWITCH |

 no switchport

 no ip address

 no ip redirects

 no ip unreachables

 no ip proxy-arp

 channel-group 44 mode on        

!

router ospf 1

router-id 2.2.2.2

 ispf

 log-adjacency-changes

 auto-cost reference-bandwidth 100000

 area 50 nssa

 redistribute connected subnets

network 10.x.x.x  0.0.0.0 area 50

This configuration from Distribution side

interface Port-channel44

 description connected to port-channel 44

 ip address 10.x.x.x 255.255.255.252

 no ip redirects

 no ip unreachables

 no ip proxy-arp

 ip ospf 1 area 50

 bfd interval 750 min_rx 750 multiplier 3

interface TenGigabitEthernet1/2/3

 description To | SWITCH

 no switchport

 no ip address

 channel-group 44 mode on

!
interface TenGigabitEthernet2/2/3

 description To | SWITCH |

 no switchport

 no ip address

 channel-group 44 mode on

!
router ospf 1

 router-id 5.5.5.5

 ispf

 auto-cost reference-bandwidth 100000

 area 50 nssa

 redistribute connected subnets

distribution model: WS-C4507R+E

IOS:cat4500e-universalk9.SPA.03.05.00.E.152-1.E.bin

CORE MODEL: WS-C6506-E

CORE: s2t54-ipservicesk9-mz.SPA.122-50.SY.bin

[coolj_212]

I configure the 4 area in the same configuration but 3 areas are working Fine and 1 area I have issue with this area also I get observe one issue the memory so I chick the memory this morning

show processes memory sorted

System memory  : 2008840K total, 1205181K used, 803659K free, 88312K kernel reserved

Then after one hour I chick this is was the output

show processes memory sorted

System memory  : 2008840K total, 1209284K used, 799556K free, 88312K kernel reserved

I Did This trablshouting when the issue happen yesterday I found in the debug

*Sep 27 21:42:47.314: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O  Free: 9096  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

 -Process= "Spanning Tree", ipl= 0, pid= 201

-Traceback= 1#49176b00b95a50f3145e3825de17d470  :1F6A5000+2CFD660 :1F6A5000+2CFDA24 :1F6A5000+4160EA0 :1F6A5000+4168078 :1F6A5000+16F0E3C :1F6A5000+16EFE7C :1F6A5000+16EB6CC :1F6A5000+3F97448 :1F6A5000+3F97BF8 :1F6A5000+3F61CC4

*Sep 27 21:43:17.384: %SYS-2-MALLOCFAIL: Memory allocation of 276 bytes failed from 0x20D95E38, alignment 8

Pool: I/O  Free: 9096  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

 -Process= "Spanning Tree", ipl= 0, pid= 201

 Hope this information help