Hello Experts ,
>> Device uptime suggests there was no reboot
ABCSwitch uptime is 28 weeks, 13 hours, 50 minutes
System returned to ROM by power-on
System restarted at 13:09:45 UTC Mon Aug 5 2013
System image file is "flash:c2950-i6k2l2q4-mz.121-22.EA12.bin"
>> But observed logs mentioning Memory Allocation Failure for TCP Protocol Process ( Process ID 43) due to Memory Fragmentation
003943: Feb 18 02:14:27.393 UTC: %SYS-2-MALLOCFAIL: Memory allocation of 36000 bytes failed from 0x801E876C, alignment 0
Pool: Processor Free: 120384 Cause: Memory fragmentation
Alternate Pool: I/O Free: 682800 Cause: Memory fragmentation
-Process= "TCP Protocols", ipl= 0, pid= 43
-Traceback= 801C422C 801C9ED0 801C5264 801E8774 801E4CDC 801D9A8C 8022E324 8022E4BC
003944: Feb 18 02:14:27.397 UTC: %SYS-2-CFORKMEM: Process creation of TCP Command failed (no memory).
-Process= "TCP Protocols", ipl= 0, pid= 43
-Traceback= 801E4D54 801D9A8C 8022E324 8022E4BC
According to Cisco documentation for Troubleshooting Memory issues on Cisco IOS 12.1 (http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-software-releases-121-mainline/6507-mallocfail.html#tshoot4 ), which suggests the TCP Protocols Process could not be started due to Memory being fragmented
Memory Fragmentation Problem or Bug
This situation means that a process has consumed a large amount of processor memory and then released most or all of it, leaving fragments of memory still allocated either by this process, or by other processes that allocated memory during the problem. If the same event occurs several times, the memory may fragment into very small blocks, to the point where all processes requiring a larger block of memory cannot get the amount of memory that they need. This may affect router operation to the extent that you cannot connect to the router and get a prompt if the memory is badly fragmented.
This problem is characterized by a low value in the "Largest" column (under 20,000 bytes) of the show memory command, but a sufficient value in the "Freed" column (1MB or more), or some other wide disparity between the two columns. This may happen when the router gets very low on memory, since there is no defragmentation routine in the IOS.
If you suspect memory fragmentation, shut down some interfaces. This may free the fragmented blocks. If this works, the memory is behaving normally, and all you have to do is add more memory. If shutting down interfaces doesn't help, it may be a bug. The best course of action is to contact your Cisco support representative with the information you have collected.
>>Further TCP -3- FORKFAIL logs were seen
003945: Feb 18 02:14:27.401 UTC: %TCP-3-FORKFAIL: Failed to start a process to negotiate options.
-Traceback= 8022E33C 8022E4BC
003946: Feb 18 02:14:27.585 UTC: %TCP-3-FORKFAIL: Failed to start a process to negotiate options.
-Traceback= 8022E33C 8022E4BC
003947: Feb 18 02:14:27.761 UTC: %TCP-3-FORKFAIL: Failed to start a process to negotiate options.
-Traceback= 8022E33C 8022E4BC
003948: Feb 18 02:14:27.929 UTC: %TCP-3-FORKFAIL: Failed to start a process to negotiate options.
-Traceback= 8022E33C 8022E4BC
003949: Feb 18 02:14:29.149 UTC: %TCP-3-FORKFAIL: Failed to start a process to negotiate options.
-Traceback= 8022E33C 8022E4BC
According to Error Explanation from Cisco Documentation (http://www.cisco.com/c/en/us/td/docs/ios/12_2sx/system/messages/122sxsms/sm2sx09.html#wp1022051)
suggests the TCP handles from a client could not be created or initialized
Error Message %TCP-3-FORKFAIL: Failed to start a process to negotiate options.
Explanation The system failed to create a process to handle requests from a client. This condition could be caused by insufficient memory.
Recommended Action Reduce other system activity to ease memory demands.
But I am still not sure about the exact root cause is as
1.The GET/GETNEXT / GET BULK messages from SNMP Manager (Here, IBM Tivoli Netcool ) uses default SNMP Port 161 which is
UDP and not TCP
2. If its ICMP Polling failure from IBM Tivoli Netcool , ICMP is Protocol Number 1 in Internet Layer of TCP/IP Protocol Suite and TCP is Protocol Number 6 in the Transport Layer of TCP/IP Protocol Suite .
So I am still not sure how TCP Protocol Process Failure could have caused ICMP Timeout . Please help !
Could you please help me on what TCP Protocol Process handles in a Cisco Switch ?
Regards,
Anup
Regards,
Anup
