High CPU due to TCP Listener Process

viveksrivastava · ‎05-17-2012

Hi everyone,

We are running a MPLS network consisting 70+ routers of 7600series.

P - 7606-S/RSP720/12.2(33)SRD3

PE - 7604/SUP32/12.2(33)SDR3

I am getting high CPU in one of my P router due to TCP Listener process.

xxxx#sh processes cpu sorted | exc 0.00%

CPU utilization for five seconds: 80%/0%; one minute: 72%; five minutes: 71%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

70 821828228 554757691 1481 75.37% 66.04% 65.37% 0 TCP Listener

503 48722596 412364583 118 1.99% 1.58% 1.57% 0 RSVP

203 2433855362470018429 0 0.87% 0.86% 0.84% 0 IP Input

Earlier also I have pafes same problem and tried to troublshoot on Cisco tools but have't got anything. So after restart, issue resolved. But this time can anyone help to find out the actual RCA.

Regards,

Vivek

amdutta · ‎05-20-2012

HI Vivek,

Can you paste the output of show stack 70 please?

Regards

viveksrivastava · ‎05-20-2012

Hi Amit,

Pls see the below o/p.

BHA_PE1#sh stacks 70

Process 70: IPC ISSU Versioning Process

Stack segment 0x52C2C9B0 - 0x52C2E120

FP: 0x52C2E060, RA: 0x414B7258

FP: 0x52C2E090, RA: 0x41BAA2B0

FP: 0x52C2E0E8, RA: 0x4146E694

FP: 0x52C2E100, RA: 0x4146E680

Regards,

Vivek

viveksrivastava · ‎05-20-2012

Hi Amit,

Pls ignore the previous o/p. Correct o/p is given below:

AUR_COR#sh stack 70

Process 70: TCP Listener

Stack segment 0x12E6274C - 0x12E63EBC

FP: 0x12E63E78, RA: 0x9478650

FP: 0x12E63EA8, RA: 0x8CA40C0

FP: 0x12E63EB0, RA: 0xA7E8918

FP: 0x0, RA: 0xA7DE558

Regards,

Vivek

amdutta · ‎05-24-2012

HI Viviek,

sorry for the late reply, can you provide show version please from the P router?

Regards

viveksrivastava · ‎05-24-2012

Hi Amit,

Pls see the below o/p. Requesting for some conclusion ASAP.

Regards,

Vivek

------------------ show version ------------------

Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:19 by prod_rel_team

ROM: System Bootstrap, Version 12.2(33r)SRD5, RELEASE SOFTWARE (fc1)
BOOTLDR: Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)

AUR_COR uptime is 6 weeks, 16 hours, 45 minutes
Uptime for this control processor is 6 weeks, 16 hours, 44 minutes
System returned to ROM by reload (SP by reload)
System restarted at 22:53:29 IST Wed Apr 11 2012
System image file is "sup-bootdisk:c7600rsp72043-advipservicesk9-mz.122-33.SRD3.bin"
Last reload type: Normal Reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco CISCO7606-S (M8500) processor (revision 1.1) with 1835008K/131072K bytes of memory.
Processor board ID FOX1326GF50
BASEBOARD: RSP720
CPU: MPC8548_E, Version: 2.0, (0x80390020)
CORE: E500, Version: 2.0, (0x80210020)
CPU:1200MHz, CCB:400MHz, DDR:200MHz,
L1: D-cache 32 kB enabled
I-cache 32 kB enabled

Last reset from power-on
1 controller (20 GigabitEthernet).
1 Virtual Ethernet interface
44 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
3964K bytes of non-volatile configuration memory.

507024K bytes of Internal ATA PCMCIA card (Sector size 512 bytes).
Configuration register is 0x2102

amdutta · ‎05-24-2012

Hi Vivek. RA points to fairly generic call functions which will not help us.

For some reason alot of TCP ports are open on the router which is casng TCP listener process to go high. Normally if this is a SNAT router doing PAT which high traffic would be an expected scenario but a P mpls router with high TCP port opened is very odd.

You can try checking with sh tcp brie and seeing how many TCP ports ar eopen, it will show you the far end ip's and ports as well. You can try and narrow it down from there, if some unknown devices or unexpoected ports are open.

Recommending that you open a case with TAC or HTTS (if you are a HTTS customer) becaus ethis would need deep dive.

Best Regards