Solved: 4500X High CPU and OSPF dropping

KGrev · ‎08-18-2023

Hi,

I have a 4500X that is dropping ospf neighbors occasionally. This behavior just sort of started one day. It is dropping opsf to its 6 neighbors intermittently about 1 neighbor every few minutes. When I run debug on ospf I can see timers hitting.

Version - WS-C4500X-32 2 slot switch cat4500e-universalk9.SPA.03.11.04.E.152-7.E4.bin

Aug 18 17:42:09.927 UTC: OSPF-10 ADJ Vl189: Send with youngest Key 1
Aug 18 17:42:10.438 UTC: OSPF-10 ADJ Vl188: Send with youngest Key 1

Aug 18 18:34:06.945 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:07.337 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:07.730 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:07.731 UTC: OSPF-10 HELLO Vl189: Send hello to 224.0.0.5 area 0 from 10.2.5.41
Aug 18 18:34:07.731 UTC: OSPF-10 PAK : Vl189: OUT: 10.2.5.41->224.0.0.5: ver:2 type:1 len:48 rid:10.2.5.130 area:0.0.0.0 chksum:0 auth:2 keyid:1 seq:0x6223
Aug 18 18:34:07.806 UTC: OSPF-10 PAK : Vl188: IN: 10.2.5.34->224.0.0.5: ver:2 type:1 len:48 rid:10.2.21.138 area:0.0.0.0 chksum:0 auth:2 keyid:1 seq:0x64DF
Aug 18 18:34:07.807 UTC: OSPF-10 HLOIN: input packet - Hello Vlan188 (0)
Aug 18 18:34:07.807 UTC: OSPF-10 HELLO Vl188: Rcv hello from 10.2.21.138 area 0 10.2.5.34
Aug 18 18:34:07.953 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:08.059 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:08.462 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:08.751 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:08.951 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:09.951 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:10.446 UTC: OSPF-10 PAK : Vl189: IN: 10.2.5.42->224.0.0.5: ver:2 type:1 len:48 rid:10.2.21.142 area:0.0.0.0 chksum:0 auth:2 keyid:1 seq:0x64DF
Aug 18 18:34:10.446 UTC: OSPF-10 HLOIN: input packet - Hello Vlan189 (0)
Aug 18 18:34:10.446 UTC: OSPF-10 HELLO Vl189: Rcv hello from 10.2.21.142 area 0 10.2.5.42
Aug 18 18:34:10.953 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:11.151 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:11.952 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:12.951 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:13.039 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:13.039 UTC: OSPF-10 HELLO Vl188: Send hello to 224.0.0.5 area 0 from 10.2.5.33
Aug 18 18:34:13.039 UTC: OSPF-10 PAK : Vl188: OUT: 10.2.5.33->224.0.0.5: ver:2 type:1 len:48 rid:10.2.5.130 area:0.0.0.0 chksum:0 auth:2 keyid:1 seq:0x6223
Aug 18 18:34:13.957 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:14.963 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:15.981 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:16.981 UTC: OSPF-10 RTRTM: timer expiration - TRAP_RATE_LIMIT_TIMER (0)
Aug 18 18:34:17.252 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:17.568 UTC: OSPF-10 PAK : Vl188: IN: 10.2.5.34->224.0.0.5: ver:2 type:1 len:48 rid:10.2.21.138 area:0.0.0.0 chksum:0 auth:2 keyid:1 seq:0x64DF
Aug 18 18:34:17.570 UTC: OSPF-10 HLOIN: input packet - Hello Vlan188 (0)
Aug 18 18:34:17.570 UTC: OSPF-10 HELLO Vl188: Rcv hello from 10.2.21.138 area 0 10.2.5.34
Aug 18 18:34:17.575 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)
Aug 18 18:34:17.667 UTC: OSPF-10 HLOTM: timer expiration - IF_HELLO_TIMER (0)

Looking at CPU utilization I could see it spiking to the 90%'s. I reloaded the switch to try to clear processes.

SWITCH#show processes cpu
Core 0: CPU utilization for five seconds: 5%; one minute: 29%; five minutes: 30%
Core 1: CPU utilization for five seconds: 91%; one minute: 65%; five minutes: 64%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 923 876 1054 0.00 0.00 0.00 0 init
2 18 432 41 0.00 0.00 0.00 0 kthreadd
3 18578 2614712 7 0.00 0.00 0.00 0 migration/0
4 0 3 3 0.00 0.00 0.00 0 sirq-high/0
5 2025466 112598644 5 0.00 0.00 0.00 0 sirq-timer/0
6 0 8 2 0.00 0.00 0.00 0 sirq-net-tx/0
7 1406132 42990322 32 0.00 0.03 0.04 0 sirq-net-rx/0
8 5438 214846 25 0.00 0.00 0.00 0 sirq-block/0
9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo
10 996 293944 3 0.00 0.00 0.02 0 sirq-tasklet/0
11 3687955 778230125 3 0.00 0.00 0.00 0 sirq-sched/0
12 0 20 2 0.00 0.00 0.00 0 sirq-hrtimer/0
13 3903030 112598645 3 0.00 0.00 0.00 0 sirq-rcu/0
14 0 20 3 0.00 0.00 0.00 0 stopper/0
15 190137 24658124 7 0.00 0.00 0.00 0 desched/0
16 10483 1473163 7 0.00 0.00 0.00 0 migration/1
17 0 23 3 0.00 0.00 0.00 0 stopper/1
18 0 17 0 0.00 0.00 0.00 0 sirq-high/1
19 631400 112598644 2 0.00 0.00 0.00 0 sirq-timer/1
20 38610 10363495 3 0.00 0.00 0.00 0 sirq-net-tx/1
21 2720872 85957893 31 0.00 0.04 0.00 0 sirq-net-rx/1
22 948 37897 25 0.00 0.00 0.00 0 sirq-block/1
23 0 22 0 0.00 0.00 0.00 0 sirq-block-iopo
24 1607 590894 2 0.00 0.00 0.00 0 sirq-tasklet/1
25 1835802 128924256 2 0.00 0.00 0.00 0 sirq-sched/1
26 0 33 2 0.00 0.00 0.00 0 sirq-hrtimer/1
27 3853432 112598645 2 0.00 0.00 0.00 0 sirq-rcu/1
28 88752 12647646 7 0.00 0.00 0.00 0 desched/1
29 1763908 93892815 18 0.00 0.00 0.00 0 events/0
30 381917 93513909 50 0.00 0.00 0.00 0 events/1
31 0 30 0 0.00 0.00 0.00 0 cpuset
32 24 1825 13 0.00 0.00 0.00 0 khelper
35 8 43 200 0.00 0.00 0.00 0 netns
36 0 35 0 0.00 0.00 0.00 0 async/mgr
218 73507 9344405 7 0.00 0.00 0.00 0 sync_supers
220 3477 187663 18 0.00 0.00 0.00 0 bdi-default
222 5973 186992 31 0.00 0.00 0.00 0 kblockd/0
223 944 30578 30 0.00 0.00 0.00 0 kblockd/1
237 1 63 26 0.00 0.00 0.00 0 khubd
240 0 39 0 0.00 0.00 0.00 0 kseriod
242 3 44 69 0.00 0.00 0.00 0 kmmcd
266 0 57 3 0.00 0.00 0.00 0 rpciod/0
267 0 57 2 0.00 0.00 0.00 0 rpciod/1
299 0 43 0 0.00 0.00 0.00 0 kswapd0
300 0 44 0 0.00 0.00 0.00 0 aio/0
301 0 45 0 0.00 0.00 0.00 0 aio/1
302 0 46 0 0.00 0.00 0.00 0 nfsiod
304 0 47 0 0.00 0.00 0.00 0 xfs_mru_cache
305 0 48 0 0.00 0.00 0.00 0 xfslogd/0
306 0 49 0 0.00 0.00 0.00 0 xfslogd/1
307 0 50 0 0.00 0.00 0.00 0 xfsdatad/0
308 0 51 0 0.00 0.00 0.00 0 xfsdatad/1
309 0 52 0 0.00 0.00 0.00 0 xfsconvertd/0
310 0 53 0 0.00 0.00 0.00 0 xfsconvertd/1
311 0 54 0 0.00 0.00 0.00 0 crypto/0
312 0 55 0 0.00 0.00 0.00 0 crypto/1
508 0 61 0 0.00 0.00 0.00 0 mtdblockd
575 0 62 0 0.00 0.00 0.00 0 usbhid_resumer
590 0 74 0 0.00 0.00 0.00 0 deferwq
646 112 1048 107 0.00 0.00 0.00 0 udevd
788 187 5689 32 0.00 0.00 0.00 0 loop1
812 0 72 1 0.00 0.00 0.00 0 loop2
836 386 8319 46 0.00 0.00 0.00 0 loop3
1816 86 821 105 0.00 0.00 0.00 0 dbus-daemon
2033 0 68 0 0.00 0.00 0.00 0 scsi_eh_0
2034 9542 1010901 9 0.00 0.00 0.00 0 usb-storage
2171 69 744 93 0.00 0.00 0.00 0 klogd
2183 3 706 4 0.00 0.00 0.00 0 portmap
2253 0 685 1 0.00 0.00 0.00 0 sshd
2284 4 698 7 0.00 0.00 0.00 0 xinetd
2288 6 971 6 0.00 0.00 0.00 0 udevd
2328 1 680 2 0.00 0.00 0.00 0 xinetd
2358 0 72 0 0.00 0.00 0.00 0 lockd
2359 103 15641 6 0.00 0.00 0.00 0 nfsd
2360 86 15642 5 0.00 0.00 0.00 0 nfsd
2361 87 15643 5 0.00 0.00 0.00 0 nfsd
2362 84 15644 5 0.00 0.00 0.00 0 nfsd
2363 89 15645 5 0.00 0.00 0.00 0 nfsd
2364 83 15646 5 0.00 0.00 0.00 0 nfsd
2365 87 15647 5 0.00 0.00 0.00 0 nfsd
2366 85 15647 5 0.00 0.00 0.00 0 nfsd
2372 0 701 0 0.00 0.00 0.00 0 rpc.mountd
2671 2 993 2 0.00 0.00 0.00 0 platform.mount
2675 1 993 1 0.00 0.00 0.00 0 inotifywait
2857 1 802 2 0.00 0.00 0.00 0 sh
2860 1 803 2 0.00 0.00 0.00 0 inotifywait
3712 0 82 0 0.00 0.00 0.00 0 edac-poller
3721 0 970 0 0.00 0.00 0.00 0 udevd
3738 0 83 0 0.00 0.00 0.00 0 vsi work/0
3739 0 84 0 0.00 0.00 0.00 0 vsi work/1
4185 553609 56043382 9 0.00 0.00 0.00 0 watchdog
4361 2397858 22155184 4373 0.08 0.09 0.08 0 sh
4697 4232503 112071057 76 0.00 0.01 0.02 0 system_mgr
4916 853324 61648325 13 0.00 0.00 0.00 1088 slproc
4951 2527286 117675859 21 0.00 0.01 0.03 1088 kstack_init
4954 201032 124021550 7 0.00 0.00 0.01 1088 platformmgr
4957 1299933 342558909 41 0.00 0.02 0.04 0 pdsvslifshim
4960 850040 61640618 13 0.00 0.00 0.00 0 ns_oir_proxy
4961 848191 61643618 13 0.00 0.00 0.00 0 profiled
4962 4229258 5243837 1625 0.00 0.01 0.04 0 oom_poll.sh
4963 901936 61647664 14 0.00 0.00 0.00 0 obfld
4964 837058 61643616 13 0.00 0.00 0.00 0 consoled
4965 830646 61643636 13 0.00 0.00 0.00 0 system_mgr_cli
4966 4286788 360950210 35 0.00 0.02 0.00 0 plogd
4967 845737 61643667 13 0.00 0.00 0.00 0 psdprov
4968 3433294 155130932 13 0.00 0.01 0.01 0 pdsd
4969 2906560 66725974 43 0.00 0.09 0.01 0 osinfo-provider
4970 843462 61643672 13 0.00 0.00 0.00 0 oscore_p
4971 1676959 66806841 25 0.00 0.00 0.00 0 netd
4972 1350155 112015656 165 0.00 0.00 0.01 0 mem_mgmt
4973 2820328 123238337 22 0.00 0.01 0.02 0 mgmte_tap
4974 2058984 63860601 166 0.00 0.01 0.01 0 licensed
4975 853387 61664368 13 0.00 0.00 0.00 0 installer
4976 3327197 137024766 134 0.24 0.19 0.20 0 ffm
4977 2271428 72160847 90 0.00 0.92 0.14 0 eicored
4978 880390 63514150 13 0.00 0.00 0.00 0 dtmgr
4979 1319901 117644439 303 0.04 0.02 0.04 0 cpumemd
4980 2344026 117869535 19 0.00 0.00 0.00 0 liin_tap
4982 871415 61690117 14 0.00 0.00 0.00 0 cli_agent
4983 1969582 298900461 6 0.00 0.21 0.40 0 snmp_subagent
4984 1740587 117680182 14 0.00 0.00 0.00 0 erm_p
4985 1494693 117680497 12 0.00 0.00 0.00 0 ngdumper_provid
4986 1498208 117680509 12 0.00 0.00 0.00 0 sysinfo_p
5004 296 10880 27 0.00 0.00 0.00 0 vman
5006 851642 61639189 13 0.00 0.00 0.00 0 fs_cmd
5019 938252 67255737 13 0.00 0.00 0.00 0 licenseagentd
5020 855688 61645018 13 0.00 0.00 0.00 0 iifd
5021 1564357 308794632 394 47.8 47.0 46.6 0 iosd
5022 389330 84770708 55 0.00 0.01 0.03 34816 ha_mgr
5589 1562219 56036406 27 0.00 0.00 0.02 34816 libvirtd
6541 15 992 15 0.00 0.00 0.00 34816 epc_sm_liaison
6561 1 997 1 0.00 0.00 0.00 0 tail
6577 0 994 0 0.00 0.00 0.00 0 script
6582 4 994 4 0.00 0.00 0.00 34816 sh
6588 14 504 28 0.00 0.00 0.00 34816 pds-kc-1
6874 215 1002 215 0.00 0.00 0.00 34816 rollback_timer.
6967 1 989 1 0.00 0.00 0.00 0 rollback_timer.
6970 50 989 50 0.00 0.00 0.00 34816 xlogger
6973 2 994 2 0.00 0.00 0.00 0 inotifywait
20323 114 958 119 0.00 0.00 0.00 34816 nova_sntp.sh
20440 0 956 0 0.00 0.00 0.00 0 nova_sntp.sh
20442 49 956 51 0.00 0.00 0.00 0 xlogger
20444 2 958 2 0.00 0.00 0.00 0 sntp
26404 3 576 5 0.00 0.00 0.00 0 flush-8:0
29554 1 5243838 0 0.00 0.00 0.00 0 sleep
29560 1 22155185 0 0.00 0.00 0.00 0 sleep

SWITCH# show memory
System memory : 3762752K total, 979332K used, 2783420K free, 431552K kernel reserved
Lowest(b) : 2233885008

Total(K) Used(K) Free(K)
Process 2816576 629145 2187432
Config 946176 350188 595988

----------------------------After reload----------------------------------------

Core 0: CPU utilization for five seconds: 19%; one minute: 29%; five minutes: 29%
Core 1: CPU utilization for five seconds: 51%; one minute: 50%; five minutes: 61%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 947 827 1145 0.00 0.00 0.00 0 init
2 1 84 18 0.00 0.00 0.00 0 kthreadd
3 8 1260 6 0.00 0.00 0.00 0 migration/0
4 0 3 2 0.00 0.00 0.00 0 sirq-high/0
5 1813 327960 5 0.00 0.00 0.00 0 sirq-timer/0
6 4 988 4 0.00 0.00 0.00 0 sirq-net-tx/0
7 167 5515 30 0.00 0.01 0.00 0 sirq-net-rx/0
8 12 481 25 0.00 0.00 0.00 0 sirq-block/0
9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo
10 2 617 3 0.00 0.00 0.00 0 sirq-tasklet/0
11 418 123905 3 0.00 0.00 0.00 0 sirq-sched/0
12 0 13 1 0.00 0.00 0.00 0 sirq-hrtimer/0
13 1176 327968 3 0.00 0.00 0.00 0 sirq-rcu/0
14 0 20 3 0.00 0.00 0.00 0 stopper/0
15 22 3844 5 0.00 0.00 0.00 0 desched/0
16 6 991 6 0.00 0.00 0.00 0 migration/1
17 0 23 2 0.00 0.00 0.00 0 stopper/1
18 0 17 0 0.00 0.00 0.00 0 sirq-high/1
19 1051 327941 3 0.00 0.00 0.00 0 sirq-timer/1
20 0 49 2 0.00 0.00 0.00 0 sirq-net-tx/1
21 130 4029 32 0.00 0.01 0.00 0 sirq-net-rx/1
22 9 207 48 0.00 0.00 0.00 0 sirq-block/1
23 0 22 0 0.00 0.00 0.00 0 sirq-block-iopo
24 1 525 3 0.00 0.00 0.00 0 sirq-tasklet/1
25 238 76560 3 0.00 0.00 0.00 0 sirq-sched/1
26 0 28 1 0.00 0.00 0.00 0 sirq-hrtimer/1
27 1106 327949 3 0.00 0.00 0.00 0 sirq-rcu/1
28 17 3214 5 0.00 0.00 0.00 0 desched/1
29 212 3015 70 0.00 0.00 0.00 0 events/0
30 78 2383 33 0.00 0.00 0.00 0 events/1
31 0 30 0 0.00 0.00 0.00 0 cpuset
32 24 1825 13 0.00 0.00 0.00 0 khelper
35 8 43 200 0.00 0.00 0.00 0 netns
36 0 35 0 0.00 0.00 0.00 0 async/mgr
218 2 261 9 0.00 0.00 0.00 0 sync_supers
220 0 41 3 0.00 0.00 0.00 0 bdi-default
222 1 149 7 0.00 0.00 0.00 0 kblockd/0
223 0 89 4 0.00 0.00 0.00 0 kblockd/1
237 1 63 26 0.00 0.00 0.00 0 khubd
240 0 39 0 0.00 0.00 0.00 0 kseriod
242 3 44 69 0.00 0.00 0.00 0 kmmcd
266 0 71 4 0.00 0.00 0.00 0 rpciod/0
267 0 43 0 0.00 0.00 0.00 0 rpciod/1
299 0 43 0 0.00 0.00 0.00 0 kswapd0
300 0 44 0 0.00 0.00 0.00 0 aio/0
301 0 45 0 0.00 0.00 0.00 0 aio/1
302 0 46 0 0.00 0.00 0.00 0 nfsiod
304 0 47 0 0.00 0.00 0.00 0 xfs_mru_cache
305 0 48 0 0.00 0.00 0.00 0 xfslogd/0
306 0 49 0 0.00 0.00 0.00 0 xfslogd/1
307 0 50 0 0.00 0.00 0.00 0 xfsdatad/0
308 0 51 0 0.00 0.00 0.00 0 xfsdatad/1
309 0 52 0 0.00 0.00 0.00 0 xfsconvertd/0
310 0 53 0 0.00 0.00 0.00 0 xfsconvertd/1
311 0 54 0 0.00 0.00 0.00 0 crypto/0
312 0 55 0 0.00 0.00 0.00 0 crypto/1
508 0 61 0 0.00 0.00 0.00 0 mtdblockd
575 0 62 0 0.00 0.00 0.00 0 usbhid_resumer
590 0 74 0 0.00 0.00 0.00 0 deferwq
646 112 1066 105 0.00 0.00 0.00 0 udevd
786 183 5593 32 0.00 0.00 0.00 0 loop1
810 0 71 1 0.00 0.00 0.00 0 loop2
834 397 8314 47 0.00 0.00 0.00 0 loop3
1813 85 817 104 0.00 0.00 0.00 0 dbus-daemon
2022 0 68 0 0.00 0.00 0.00 0 scsi_eh_0
2024 16 2682 6 0.00 0.00 0.00 0 usb-storage
2168 69 752 91 0.00 0.00 0.00 0 klogd
2180 3 702 5 0.00 0.00 0.00 0 portmap
2264 6 988 6 0.00 0.00 0.00 0 udevd
2308 0 681 1 0.00 0.00 0.00 0 sshd
2322 3 684 5 0.00 0.00 0.00 0 xinetd
2328 1 675 2 0.00 0.00 0.00 0 xinetd
2431 0 72 0 0.00 0.00 0.00 0 lockd
2432 0 74 0 0.00 0.00 0.00 0 nfsd
2433 0 75 0 0.00 0.00 0.00 0 nfsd
2434 0 76 0 0.00 0.00 0.00 0 nfsd
2435 0 77 0 0.00 0.00 0.00 0 nfsd
2436 0 78 0 0.00 0.00 0.00 0 nfsd
2437 0 79 0 0.00 0.00 0.00 0 nfsd
2438 0 80 0 0.00 0.00 0.00 0 nfsd
2439 0 80 0 0.00 0.00 0.00 0 nfsd
2482 0 699 0 0.00 0.00 0.00 0 rpc.mountd
2755 2 1009 2 0.00 0.00 0.00 0 platform.mount
2762 1 1009 1 0.00 0.00 0.00 0 inotifywait
2954 1 801 2 0.00 0.00 0.00 0 sh
2957 1 802 2 0.00 0.00 0.00 0 inotifywait
3104 3 255 13 0.00 0.00 0.00 0 flush-8:0
3809 0 82 0 0.00 0.00 0.00 0 edac-poller
3815 0 986 0 0.00 0.00 0.00 0 udevd
3835 0 83 0 0.00 0.00 0.00 0 vsi work/0
3836 0 84 0 0.00 0.00 0.00 0 vsi work/1
4282 92 2985 31 0.00 0.00 0.00 0 watchdog
4425 486 1617 300 0.00 0.00 0.03 0 sh
4794 385 4263 90 0.00 0.02 0.04 0 system_mgr
5013 273 12777 21 0.00 0.00 0.00 1088 slproc
5048 84 6001 14 0.00 0.00 0.00 1088 kstack_init
5052 276 26264 10 0.00 0.01 0.02 1088 platformmgr
5054 713 19348 36 0.00 0.02 0.03 0 pdsvslifshim
5056 136 5455 25 0.00 0.00 0.00 0 ns_oir_proxy
5057 124 8383 14 0.00 0.00 0.00 0 profiled
5058 185 1071 173 0.00 0.01 0.02 0 oom_poll.sh
5060 212 12507 16 0.00 0.00 0.00 0 obfld
5061 116 8361 13 0.00 0.00 0.00 0 consoled
5062 158 8385 18 0.00 0.00 0.00 0 system_mgr_cli
5063 523 15634 33 0.00 0.00 0.00 0 plogd
5064 139 8423 16 0.00 0.00 0.01 0 psdprov
5065 1694 86511 19 0.00 0.01 0.01 0 pdsd
5066 193 8417 22 0.00 0.00 0.00 0 osinfo-provider
5067 129 8484 15 0.00 0.00 0.00 0 oscore_p
5068 137 6707 20 0.00 0.00 0.04 0 netd
5069 617 8927 69 0.00 0.01 0.00 0 mem_mgmt
5070 168 7459 22 0.00 0.00 0.00 0 mgmte_tap
5071 2890 24973 115 0.08 0.08 0.25 0 licensed
5072 336 27749 12 0.00 0.00 0.23 0 installer
5073 9668 46945 205 0.16 0.20 0.66 0 ffm
5074 2314 38950 59 0.00 0.01 0.25 0 eicored
5075 236 10506 22 0.00 0.00 0.00 0 dtmgr
5076 931 8687 107 0.00 0.01 0.02 0 cpumemd
5077 137 6953 19 0.00 0.01 0.03 0 liin_tap
5081 378 9699 39 0.00 0.02 0.61 0 cli_agent
5082 582 29287 19 0.00 0.00 0.03 0 snmp_subagent
5083 148 9652 15 0.00 0.00 0.00 0 erm_p
5084 137 9608 14 0.00 0.00 0.01 0 ngdumper_provid
5085 141 9620 14 0.00 0.00 0.00 0 sysinfo_p
5092 277 10409 26 0.00 0.00 0.00 0 vman
5093 59 3984 14 0.00 0.00 0.00 0 fs_cmd
5112 371 16075 23 0.00 0.00 0.00 0 licenseagentd
5113 331 9443 35 0.00 0.00 0.00 34816 iifd
5114 1049478 801592 1309 35.0 39.7 43.1 0 iosd
5115 653 13025 50 0.00 0.06 0.07 0 ha_mgr
5696 61 2144 28 0.00 0.00 0.02 34816 libvirtd
6637 15 977 15 0.00 0.00 0.00 0 epc_sm_liaison
6658 1 979 1 0.00 0.00 0.00 0 tail
6666 0 981 0 0.00 0.00 0.00 0 script
6673 3 980 3 0.00 0.00 0.00 34816 sh
6687 0 85 0 0.00 0.00 0.00 34816 pds-kc-1
6965 214 991 216 0.00 0.00 0.00 34816 rollback_timer.
7071 1 981 1 0.00 0.00 0.00 0 rollback_timer.
7074 50 982 51 0.00 0.00 0.00 0 xlogger
7078 2 985 2 0.00 0.00 0.00 0 inotifywait
7619 106 920 115 0.00 0.00 0.00 0 nova_sntp.sh
7718 0 918 0 0.00 0.00 0.00 0 nova_sntp.sh
7719 42 918 46 0.00 0.00 0.00 0 xlogger
7723 3 923 3 0.00 0.00 0.00 0 sntp
8796 1 1071 1 0.00 0.00 0.00 0 sleep
8811 1 1617 0 0.00 0.00 0.00 0 sleep

System memory : 3762752K total, 912448K used, 2850304K free, 431552K kernel reserved
Lowest(b) : 2512289968

Total(K) Used(K) Free(K)
Process 2816576 567597 2248980
Config 946176 344852 601324

After the reload the utilization went down some but something that was present before and after the reload was high latency while pinging the loopback of the switch. Around 30-60ms latency consistently. We have a simple network here compared to higher tier networks. I am on a cisco 3850 pinging the 4500X over a fiber link with nothing in between.

Any ideas or help will be greatly appreciated

*note i will be leaving the office soon so any feedback i need to gather will be delayed. I'm very sorry.*

Peter Paluch · ‎08-21-2023

Hello @KGrev ,

Thank you very much for sharing the outputs. Before I dive deeper, one humble request: In the future, if possible, save the outputs in a file and attach the file to your post, instead of posting the output directly. When you just copy&paste the outputs, aside from them being impractically long, their formatting is lost and they are very difficult to read.

As for the CPU load, it does not appear to be abnormal (yet), and the processes consuming the most CPU time are fairly common to see there - so that does not give us a clue.

Looking at the "show ip traffic" output, however, I am surprised to see quite a large amount of received broadcasts. Do you know what application in your network could be responsible for generating a large amount of IP broadcasts? It might be that the amount of IP broadcasts is excessive and gets in the way of processing OSPF packets or ICMP pings. Hence, I would like to narrow down the cause of those IP broadcasts before we move on.

We have an option of recording the traffic that hits your supervisor into a PCAP file and have a look at it in Wireshark. With switches, we should be seeing very little traffic on the supervisor because all flows should be handled in hardware - so any extra traffic there is a potential reason to troubleshoot. This PCAP file would be very helpful but at the same time, it cannot be sanitized the way text outputs are sanitized, so it may hold sensitive information - it is up to you to decide if you first want to even try to do this control plane packet capture, and if so, whether you would like to analyze the file in Wireshark yourself, or whether you can also share it here.

Best regards,
Peter

View solution in original post

Peter Paluch · ‎08-18-2023

Hello,

We will indeed some extra outputs to understand the situation on the switch better.

Regarding the debugs, I don't think they are showing anything unexpected. They are simply indicating that certain internal timers in the OSPF process have fired, but that does not necessarily mean that some expected event did not occur.

Would it be possible to share the following outputs from your switch please?

terminal length 0
show logging
show processes cpu sorted
show processes cpu detailed process iosd sorted
show ip traffic
show ip ospf traffic
show ip ospf neighbor
show ip ospf interface
show ip ospf database database-summary

These are some good outputs to start with - here, I'm trying to see the OSPF logging messages to help me understand what was the actual reason OSPF dropped the adjacency, then I'm trying to collect the information about the processes both in IOS-XE and inside the IOSd binary to understand what might be causing the high CPU, then information about IP traffic hitting your supervisor, and finally I'd like to collect some information about the OSPF process itself.

Ideally, please collect all these outputs into a single file and share it here if possible. Of course, sanitize sensitive information but do not otherwise remove details - everything may be important.

Thank you!

Best regards,
Peter

MHM Cisco World · ‎08-18-2023

I send you message check it

Leo Laohoo · ‎08-18-2023

Post the complete output to the following commands:

sh proc cpu sort | ex 0.00
sh memory sort | ex 0.00

Peter Paluch · ‎08-19-2023

Hi Leo,

I've recently had an opportunity to help with another Cat4500 case, and I learned that here, at least the "show processes cpu" command (don't recall about "show memory") produces a list of processes from the underlying Linux operating system. However, you won't see the processes inside the IOSd blob which are the ones that we both are interested in - such as IP Input or OSPF; we will only see iosd as a single Linux process. That is why I asked for the "show processes cpu detailed process iosd sorted" output in my post that drills down into IOSd processes the usual way we were used to in old IOS.

I thought you might be interested to know : )

Best regards,
Peter

Leo Laohoo · ‎08-19-2023

Thanks for that, @Peter Paluch.

We have a few Sup7 left. Might try this out later.

KGrev · ‎08-21-2023

@Leo Laohoo @Peter Paluch @MHM Cisco World
Here are all the outputs you asked for.
Thank you for helping me through this.

SWITCH_4500X#show processes cpu sorted
Core 0: CPU utilization for five seconds: 13%; one minute: 16%; five minutes: 15%
Core 1: CPU utilization for five seconds: 14%; one minute: 12%; five minutes: 13%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5114 3452009 15490452 826 13.47 13.86 13.98 34816 iosd
5073 1012842 5864399 172 0.20 0.20 0.20 0 ffm
4425 329860 114199 2888 0.04 0.07 0.08 0 sh
5052 45120 5226584 8 0.04 0.02 0.03 1088 platformmgr
5063 60277 1513725 39 0.04 0.02 0.02 0 plogd
5069 82501 478986 172 0.04 0.03 0.04 0 mem_mgmt
1 947 827 1145 0.00 0.00 0.00 0 init
2 2 93 21 0.00 0.00 0.00 0 kthreadd
3 124 18104 6 0.00 0.00 0.00 0 migration/0
4 0 3 2 0.00 0.00 0.00 0 sirq-high/0
5 329865 59133874 5 0.00 0.00 0.00 0 sirq-timer/0
6 962 236212 4 0.00 0.00 0.00 0 sirq-net-tx/0
7 13339 419764 31 0.00 0.00 0.03 0 sirq-net-rx/0
8 43 1695 25 0.00 0.00 0.00 0 sirq-block/0
9 0 8 0 0.00 0.00 0.00 0 sirq-block-iopo
10 22 6424 3 0.00 0.00 0.00 0 sirq-tasklet/0
11 64238 18932242 3 0.00 0.00 0.00 0 sirq-sched/0
12 0 13 1 0.00 0.00 0.00 0 sirq-hrtimer/0
13 204042 59133882 3 0.00 0.00 0.00 0 sirq-rcu/0
14 0 20 3 0.00 0.00 0.00 0 stopper/0
15 761 102631 7 0.00 0.00 0.00 0 desched/0
16 100 13786 7 0.00 0.00 0.00 0 migration/1
17 0 23 2 0.00 0.00 0.00 0 stopper/1
18 0 17 0 0.00 0.00 0.00 0 sirq-high/1
19 183680 59133856 3 0.00 0.00 0.00 0 sirq-timer/1
20 0 49 2 0.00 0.00 0.00 0 sirq-net-tx/1
21 16792 545623 30 0.00 0.02 0.03 0 sirq-net-rx/1
22 12 315 39 0.00 0.00 0.00 0 sirq-block/1
23 0 22 0 0.00 0.00 0.00 0 sirq-block-iopo
24 18 6229 2 0.00 0.00 0.00 0 sirq-tasklet/1
25 68253 23219786 2 0.00 0.00 0.00 0 sirq-sched/1
26 0 28 1 0.00 0.00 0.00 0 sirq-hrtimer/1
27 172683 59133864 2 0.00 0.00 0.00 0 sirq-rcu/1
28 402 60944 6 0.00 0.00 0.00 0 desched/1
29 46597 642137 72 0.00 0.00 0.00 0 events/0
30 12909 340141 37 0.00 0.00 0.00 0 events/1
31 0 30 0 0.00 0.00 0.00 0 cpuset
32 24 1825 13 0.00 0.00 0.00 0 khelper
35 8 43 200 0.00 0.00 0.00 0 netns
36 0 35 0 0.00 0.00 0.00 0 async/mgr
218 364 39508 9 0.00 0.00 0.00 0 sync_supers
220 15 846 18 0.00 0.00 0.00 0 bdi-default
222 30 1054 29 0.00 0.00 0.00 0 kblockd/0
223 1 133 13 0.00 0.00 0.00 0 kblockd/1
237 1 63 26 0.00 0.00 0.00 0 khubd
240 0 39 0 0.00 0.00 0.00 0 kseriod
242 3 44 69 0.00 0.00 0.00 0 kmmcd
266 0 71 4 0.00 0.00 0.00 0 rpciod/0
267 0 43 0 0.00 0.00 0.00 0 rpciod/1
299 0 43 0 0.00 0.00 0.00 0 kswapd0
300 0 44 0 0.00 0.00 0.00 0 aio/0
301 0 45 0 0.00 0.00 0.00 0 aio/1
302 0 46 0 0.00 0.00 0.00 0 nfsiod
304 0 47 0 0.00 0.00 0.00 0 xfs_mru_cache
305 0 48 0 0.00 0.00 0.00 0 xfslogd/0
306 0 49 0 0.00 0.00 0.00 0 xfslogd/1
307 0 50 0 0.00 0.00 0.00 0 xfsdatad/0
308 0 51 0 0.00 0.00 0.00 0 xfsdatad/1
309 0 52 0 0.00 0.00 0.00 0 xfsconvertd/0
310 0 53 0 0.00 0.00 0.00 0 xfsconvertd/1
311 0 54 0 0.00 0.00 0.00 0 crypto/0
312 0 55 0 0.00 0.00 0.00 0 crypto/1
508 0 61 0 0.00 0.00 0.00 0 mtdblockd
575 0 62 0 0.00 0.00 0.00 0 usbhid_resumer
590 0 74 0 0.00 0.00 0.00 0 deferwq
646 112 1066 105 0.00 0.00 0.00 0 udevd
786 183 5601 32 0.00 0.00 0.00 0 loop1
810 0 71 1 0.00 0.00 0.00 0 loop2
834 397 8314 47 0.00 0.00 0.00 0 loop3
1813 85 817 104 0.00 0.00 0.00 0 dbus-daemon
2022 0 68 0 0.00 0.00 0.00 0 scsi_eh_0
2024 63 7970 8 0.00 0.00 0.00 0 usb-storage
2168 69 752 91 0.00 0.00 0.00 0 klogd
2180 3 702 5 0.00 0.00 0.00 0 portmap
2264 6 988 6 0.00 0.00 0.00 0 udevd
2308 0 681 1 0.00 0.00 0.00 0 sshd
2322 3 684 5 0.00 0.00 0.00 0 xinetd
2328 1 675 2 0.00 0.00 0.00 0 xinetd
2431 0 72 0 0.00 0.00 0.00 0 lockd
2432 0 139 4 0.00 0.00 0.00 0 nfsd
2433 0 140 2 0.00 0.00 0.00 0 nfsd
2434 0 141 2 0.00 0.00 0.00 0 nfsd
2435 0 142 2 0.00 0.00 0.00 0 nfsd
2436 0 143 2 0.00 0.00 0.00 0 nfsd
2437 0 144 2 0.00 0.00 0.00 0 nfsd
2438 0 145 2 0.00 0.00 0.00 0 nfsd
2439 0 145 2 0.00 0.00 0.00 0 nfsd
2482 0 699 0 0.00 0.00 0.00 0 rpc.mountd
2755 2 1009 2 0.00 0.00 0.00 0 platform.mount
2762 1 1009 1 0.00 0.00 0.00 0 inotifywait
2954 1 801 2 0.00 0.00 0.00 0 sh
2957 1 802 2 0.00 0.00 0.00 0 inotifywait
3809 0 82 0 0.00 0.00 0.00 0 edac-poller
3815 0 986 0 0.00 0.00 0.00 0 udevd
3835 0 83 0 0.00 0.00 0.00 0 vsi work/0
3836 0 84 0 0.00 0.00 0.00 0 vsi work/1
4282 2637 238201 11 0.00 0.00 0.02 0 watchdog
4794 38355 474631 80 0.00 0.01 0.03 0 system_mgr
5013 4128 271475 15 0.00 0.00 0.00 1088 slproc
5048 11852 499843 23 0.00 0.00 0.00 1088 kstack_init
5054 62531 1355342 46 0.00 0.03 0.04 0 pdsvslifshim
5056 3956 264132 14 0.00 0.00 0.00 0 ns_oir_proxy
5057 3910 267062 14 0.00 0.01 0.08 0 profiled
5058 11191 21239 526 0.00 0.03 0.04 0 oom_poll.sh
5060 4204 271184 15 0.00 0.01 0.00 0 obfld
5061 3931 267040 14 0.00 0.01 0.05 0 consoled
5062 3973 267065 14 0.00 0.01 0.08 0 system_mgr_cli
5064 3903 267102 14 0.00 0.02 0.10 0 psdprov
5065 116988 7330528 15 0.00 0.02 0.03 0 pdsd
5066 4021 267128 15 0.00 0.02 0.10 0 osinfo-provider
5067 4053 267162 15 0.00 0.01 0.08 0 oscore_p
5068 7548 284555 26 0.00 0.00 0.00 0 netd
5070 20038 627072 31 0.00 0.00 0.03 0 mgmte_tap
5071 47709 288774 165 0.00 0.00 0.03 0 licensed
5072 4083 286436 14 0.00 0.01 0.08 0 installer
5074 6397 298402 21 0.00 0.01 0.08 0 eicored
5075 4194 277032 15 0.00 0.21 0.93 0 dtmgr
5076 158230 502385 314 0.00 0.03 0.04 0 cpumemd
5077 11443 505170 22 0.00 0.02 0.02 0 liin_tap
5081 4254 268775 15 0.00 0.02 0.10 0 cli_agent
5082 4509 1355219 3 0.00 0.00 0.01 0 snmp_subagent
5083 8154 503502 16 0.00 0.00 0.05 0 erm_p
5084 6976 503459 13 0.00 0.00 0.01 0 ngdumper_provid
5085 6788 503475 13 0.00 0.01 0.05 0 sysinfo_p
5092 282 10412 27 0.00 0.00 0.02 0 vman
5093 3792 262660 14 0.00 0.00 0.02 0 fs_cmd
5112 4617 298278 15 0.00 0.01 0.08 0 licenseagentd
5113 4123 268122 15 0.00 0.01 0.08 0 iifd
5115 21829 369496 59 0.00 0.01 0.08 0 ha_mgr
5696 7511 237307 31 0.00 0.00 0.00 0 libvirtd
6637 15 977 15 0.00 0.00 0.00 34816 epc_sm_liaison
6658 1 979 1 0.00 0.00 0.00 0 tail
6666 0 981 0 0.00 0.00 0.00 0 script
6673 3 980 3 0.00 0.00 0.00 34816 sh
6687 0 85 0 0.00 0.00 0.00 34816 pds-kc-1
6965 214 991 216 0.00 0.00 0.00 34816 rollback_timer.
7071 1 981 1 0.00 0.00 0.00 34816 rollback_timer.
7074 50 982 51 0.00 0.00 0.00 0 xlogger
7078 2 985 2 0.00 0.00 0.00 0 inotifywait
7343 1 21239 0 0.00 0.00 0.00 0 sleep
7391 1 114200 0 0.00 0.00 0.00 0 sleep
7619 106 920 115 0.00 0.00 0.00 0 nova_sntp.sh
7718 0 918 0 0.00 0.00 0.00 0 nova_sntp.sh
7719 42 918 46 0.00 0.00 0.00 0 xlogger
7723 3 923 3 0.00 0.00 0.00 0 sntp
18854 11 741 15 0.00 0.00 0.00 0 flush-8:0

--------------------------------------------------------------------------------------------------------

SWITCH_4500X#show processes cpu sorted | ex 0.00
Core 0: CPU utilization for five seconds: 17%; one minute: 14%; five minutes: 15%
Core 1: CPU utilization for five seconds: 27%; one minute: 18%; five minutes: 15%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5114 3559264 15517526 825 21.82 16.05 14.71 34816 iosd
4425 330362 114333 2889 0.12 0.08 0.08 0 sh
5071 47776 289117 165 0.12 0.03 0.10 0 licensed
5052 45181 5233336 8 0.04 0.02 0.01 1088 platformmgr
5060 4209 271520 15 0.04 0.04 0.04 0 obfld
5063 60351 1515698 39 0.04 0.03 0.02 0 plogd
5073 1014084 5871940 172 0.04 0.20 0.20 0 ffm
5081 4347 269350 16 0.04 0.02 0.16 0 cli_agent

---------------------------------------------------------------------------------------------------------

SWITCH_4500X#show processes cpu detailed process iosd sorted
Core 0: CPU utilization for five seconds: 20%; one minute: 16%; five minutes: 15%
Core 1: CPU utilization for five seconds: 13%; one minute: 13%; five minutes: 13%
PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
(%) (%) (%)
5114 L 3461852 1549304 826 14.67 14.08 14.02 34816 iosd
5114 L 1 5114 2288985 7023046 0 12.19 12.12 12.11 34816 iosd
5114 L 0 7469 1146783 8431351 0 2.48 1.96 1.91 34816 iosd.fastpath
5114 L 1 7470 21781 142494 0 0.00 0.03 0.04 34816 CMI Thread
5114 L 0 7471 4103 237677 0 0.00 0.00 0.00 0 iosd.monitor
5114 L 1 7472 200 6297 0 0.00 0.00 0.00 0 iosd.aux
88 I 339878 4489633 0 9.99 8.99 9.22 0 Cat4k Mgmt HiPri
89 I 675331 2269592 0 6.77 6.44 6.44 0 Cat4k Mgmt LoPri
190 I 1608326 3310078 0 2.44 2.88 2.88 0 IP Input
276 I 2109536 1450876 0 1.11 0.55 0.88 0 IP SNMP
274 I 691464 7240666 0 0.55 0.22 0.33 0 PDU DISPATCHER
275 I 763840 7315755 0 0.33 0.11 0.22 0 SNMP ENGINE
167 I 401564 4620387 0 0.22 0.11 0.11 0 Tunnel IOSd shim DB
199 I 495628 1487874 0 0.11 0.11 0.11 0 Spanning Tree
359 I 1020 646 0 0.11 0.22 0.22 1 SSH Process
412 I 136144 5395331 0 0.11 0.00 0.00 0 PIM Process
323 I 74884 7382162 0 0.00 0.00 0.00 0 MMON MENG
189 I 75216 7075967 0 0.00 0.00 0.00 0 IP ARP Retry Ager
21 I 49288 146323 0 0.00 0.00 0.00 0 CMI IOSd task
13 I 8 23 0 0.00 0.00 0.00 0 IPC Apps Task
15 I 240 183 0 0.00 0.00 0.00 0 RF Slave Main Threa
14 I 0 2 0 0.00 0.00 0.00 0 Serial Background
12 I 0 344 0 0.00 0.00 0.00 0 WATCH_AFS
11 I 0 2 0 0.00 0.00 0.00 0 Timers
10 I 0 1 0 0.00 0.00 0.00 0 DiscardQ Background
16 I 0 10 0 0.00 0.00 0.00 0 ifIndex Receive Pro
17 I 28516 54954 0 0.00 0.00 0.00 0 IOSD ipc task
22 I 32 33 0 0.00 0.00 0.00 0 CMI IOSd task
9 I 1392 3943 0 0.00 0.00 0.00 0 Pool Manager
24 I 0 2 0 0.00 0.00 0.00 0 CMI IOSd task
25 I 0 2 0 0.00 0.00 0.00 0 CMI IOSd task
26 I 6392 238473 0 0.00 0.00 0.00 0 EPC SM Liaison Upda
8 I 365540 48490 0 0.00 0.11 0.11 0 Check heaps
18 I 0 3 0 0.00 0.00 0.00 0 IOSD chasfs task
19 I 4 8 0 0.00 0.00 0.00 0 cpf_msg_holdq_proce
30 I 3996 248223 0 0.00 0.00 0.00 0 ARP Background
20 I 108 15749 0 0.00 0.00 0.00 0 IOSD taskmonitor
32 I 0 1 0 0.00 0.00 0.00 0 AAA_SERVER_DEADTIME
23 I 0 4 0 0.00 0.00 0.00 0 CMI IOSd task
34 I 24 21 0 0.00 0.00 0.00 0 Entity MIB API
35 I 0 1 0 0.00 0.00 0.00 0 IFS Agent Manager
36 I 1548 47270 0 0.00 0.00 0.00 0 IPC Event Notifier
37 I 2436 230451 0 0.00 0.00 0.00 0 IPC Mcast Pending G
38 I 0 2 0 0.00 0.00 0.00 0 Galios IPC Bootstra
39 I 36 3941 0 0.00 0.00 0.00 0 IPC Dynamic Cache
27 I 1888 235986 0 0.00 0.00 0.00 0 EPC WS Test Pkt Sen
41 I 12004 186970 0 0.00 0.00 0.00 0 cpf_msg_rcvq_proces
42 I 7456 234047 0 0.00 0.00 0.00 0 IPC Periodic Timer
43 I 2232 230450 0 0.00 0.00 0.00 0 IPC Deferred Port C
44 I 0 1 0 0.00 0.00 0.00 0 IPC Process level s
45 I 9148 179621 0 0.00 0.00 0.00 0 IPC Seat Manager
46 I 28 4299 0 0.00 0.00 0.00 0 IPC Check Queue Tim
28 I 0 1 0 0.00 0.00 0.00 0 EPC WS Pkt Send pro
48 I 0 1 0 0.00 0.00 0.00 0 IPC Seat TX Control
49 I 380 23645 0 0.00 0.00 0.00 0 IPC Keep Alive Mess
29 I 144 495 0 0.00 0.00 0.00 0 ARP Input
51 I 0 1 0 0.00 0.00 0.00 0 IPC Session Details
7 I 2020 235986 0 0.00 0.00 0.00 0 Domain Helper - 0
53 I 0 1 0 0.00 0.00 0.00 0 SL async process
54 I 0 1 0 0.00 0.00 0.00 0 SPI PL process
31 I 0 1 0 0.00 0.00 0.00 0 CEF MIB API
33 I 0 1 0 0.00 0.00 0.00 0 Policy Manager
57 I 4 2 0 0.00 0.00 0.00 0 SPI Delayed Init
58 I 8 6 0 0.00 0.00 0.00 0 client_entity_sendi
59 I 0 1 0 0.00 0.00 0.00 0 SERIAL A'detect
40 I 8 5 0 0.00 0.00 0.00 0 IPC Session Service
61 I 2736 235985 0 0.00 0.00 0.00 0 Dynamic ARP Inspect
62 I 0 2 0 0.00 0.00 0.00 0 SMART
63 I 0 1 0 0.00 0.00 0.00 0 IOSD signals IOS cm
64 I 6328 235985 0 0.00 0.00 0.00 0 pdslib stats task
65 I 2864 236237 0 0.00 0.00 0.00 0 GraphIt
66 I 304 253 0 0.00 0.00 0.00 0 crypto sw pk proc
67 I 0 1 0 0.00 0.00 0.00 0 Critical Bkgnd
68 I 10612 159588 0 0.00 0.00 0.00 0 Net Background
69 I 42864 236255 0 0.00 0.00 0.00 0 IDB Work
70 I 160 2739 0 0.00 0.00 0.00 0 Logger
71 I 3660 235785 0 0.00 0.00 0.00 0 TTY Background
72 I 0 2 0 0.00 0.00 0.00 0 CTS CORE
73 I 0 8 0 0.00 0.00 0.00 0 SXP CORE
47 I 24 122 0 0.00 0.00 0.00 0 IPC Seat RX Control
75 I 4 82 0 0.00 0.00 0.00 0 IF-MGR event proces
76 I 12 29 0 0.00 0.00 0.00 0 PrstVbl IPC Msg Pro
77 I 0 1 0 0.00 0.00 0.00 0 VSLP ISSU receive p
78 I 0 2 0 0.00 0.00 0.00 0 Parser SPI sever
79 I 4428 46852 0 0.00 0.00 0.00 0 cpf_process_tpQ
80 I 0 1 0 0.00 0.00 0.00 0 sw_idbman_nova IIF
81 I 0 7 0 0.00 0.00 0.00 0 CTS HA
82 I 0 2 0 0.00 0.00 0.00 0 IP Admission HA Pro
83 I 0 2 0 0.00 0.00 0.00 0 Net Input
84 I 21384 50726 0 0.00 0.00 0.00 0 Compute load avgs
85 I 154812 12053 0 0.00 0.00 0.00 0 Per-minute Jobs
86 I 18392 260179 0 0.00 0.00 0.00 0 Per-Second Jobs
50 I 2656 47279 0 0.00 0.00 0.00 0 IPC Loadometer
6 I 0 2 0 0.00 0.00 0.00 0 IPC ISSU Dispatch P
5 I 0 113 0 0.00 0.00 0.00 0 Retransmission of I
90 I 1444 82232 0 0.00 0.00 0.00 0 Galios Rescheduler
91 I 188 3493 0 0.00 0.00 0.00 0 GaliosObflLogger_he
92 I 0 1 0 0.00 0.00 0.00 0 GaliosVssBfd_proces
52 I 4 10 0 0.00 0.00 0.00 0 PrstVbl
94 I 0 1 0 0.00 0.00 0.00 0 ClearDynamicMacEntr
95 I 0 1 0 0.00 0.00 0.00 0 VSLP ISSU session p
96 I 0 1 0 0.00 0.00 0.00 0 XDR RRP RF waiter
97 I 0 2 0 0.00 0.00 0.00 0 AutoQoS HA
55 I 0 1 0 0.00 0.00 0.00 0 SPI PL FC process
99 I 0 5 0 0.00 0.00 0.00 0 IP Host Track HA Pr
100 I 16 277 0 0.00 0.00 0.00 0 ARP HA
101 I 0 3 0 0.00 0.00 0.00 0 SISF HA Process
102 I 0 5 0 0.00 0.00 0.00 0 MSP HA
56 I 0 1 0 0.00 0.00 0.00 0 SPI IPC Cleanup pro
104 I 72 21 0 0.00 0.00 0.00 0 VSLP RRP event gene
105 I 60320 1475973 0 0.00 0.00 0.00 0 LMP state machine p
60 I 0 1 0 0.00 0.00 0.00 0 ARP Snoop
107 I 0 7 0 0.00 0.00 0.00 0 avc_dns_as_ha_ipc
108 I 0 1 0 0.00 0.00 0.00 0 CONSOLE helper proc
109 I 20 57 0 0.00 0.00 0.00 0 GalRedMsgProc
110 I 4 8 0 0.00 0.00 0.00 0 dbflags resync with
111 I 0 5 0 0.00 0.00 0.00 0 Metadata HA
112 I 0 7 0 0.00 0.00 0.00 0 FMD HA IPC flow con
74 I 20 66 0 0.00 0.00 0.00 0 IF-MGR control proc
114 I 0 6 0 0.00 0.00 0.00 0 DHCPC HA
115 I 0 6 0 0.00 0.00 0.00 0 DHCPD HA
116 I 0 2 0 0.00 0.00 0.00 0 Network-rf Notify N
117 I 0 8 0 0.00 0.00 0.00 0 RFS server process
118 I 8 18 0 0.00 0.00 0.00 0 RTTYS Process
119 I 4800 235984 0 0.00 0.00 0.00 0 GaliosBfd_auditPend
120 I 0 3 0 0.00 0.00 0.00 0 GaliosSysNovaPosix_
121 I 0 2 0 0.00 0.00 0.00 0 GaliosSysNovaSpiIni
122 I 3124 236027 0 0.00 0.00 0.00 0 RedStateMonitor
123 I 0 31 0 0.00 0.00 0.00 0 BACK CHECK
124 I 0 1 0 0.00 0.00 0.00 0 AggMgr Process
125 I 0 1 0 0.00 0.00 0.00 0 Transport Port Agen
126 I 8492 70834 0 0.00 0.00 0.00 0 HC Counter Timers
127 I 120 3482 0 0.00 0.00 0.00 0 SFF8472
128 I 0 1 0 0.00 0.00 0.00 0 iosv_debug_trace
129 I 44 3937 0 0.00 0.00 0.00 0 Crashinfo File Tran
130 I 0 1 0 0.00 0.00 0.00 0 Src Fltr background
131 I 0 4 0 0.00 0.00 0.00 0 EEM ED Identity
132 I 0 4 0 0.00 0.00 0.00 0 EEM ED MAT
133 I 1752 43801 0 0.00 0.00 0.00 0 EEM ED ND
134 I 0 70 0 0.00 0.00 0.00 0 SSH Event handler
135 I 1912 47166 0 0.00 0.00 0.00 0 MGMTE stats Process
136 I 0 2 0 0.00 0.00 0.00 0 worker thread
137 I 1088 118015 0 0.00 0.00 0.00 0 System polling
4 I 16 23 0 0.00 0.00 0.00 0 SpanTree Flush
139 I 0 1 0 0.00 0.00 0.00 0 EAP Test
140 I 0 2 0 0.00 0.00 0.00 0 Ethernet CFM
87 I 92 284 0 0.00 0.00 0.00 0 GaliosQuack_helper
142 I 306888 3202862 0 0.00 0.00 0.00 0 Ethernet Msec Timer
143 I 0 2 0 0.00 0.00 0.00 0 Ethernet OAM Proces
144 I 0 1 0 0.00 0.00 0.00 0 PPPOE IA
145 I 0 2 0 0.00 0.00 0.00 0 REP Topology change
146 I 0 3 0 0.00 0.00 0.00 0 Netflow-lite Export
147 I 0 1 0 0.00 0.00 0.00 0 Netflow-Lite
148 I 364 3938 0 0.00 0.00 0.00 0 GalRedConfSynPlfmCl
149 I 0 2 0 0.00 0.00 0.00 0 Dot1x Mgr Process
150 I 0 19 0 0.00 0.00 0.00 0 802.1x switch
151 I 0 2 0 0.00 0.00 0.00 0 DTP Protocol
152 I 20 3939 0 0.00 0.00 0.00 0 Port-Security
153 I 0 1 0 0.00 0.00 0.00 0 ACL Snoop (IPSG)
154 I 0 2 0 0.00 0.00 0.00 0 EFP Errd
155 I 28 51 0 0.00 0.00 0.00 0 Qos If Bw Change
156 I 171276 2362352 0 0.00 0.00 0.00 0 UDLD
157 I 0 2 0 0.00 0.00 0.00 0 IP Host Track Proce
158 I 0 2 0 0.00 0.00 0.00 0 Dual-Active Recover
159 I 0 1 0 0.00 0.00 0.00 0 Dual_active fast-he
160 I 0 1 0 0.00 0.00 0.00 0 Switch Backup Inter
161 I 24 3938 0 0.00 0.00 0.00 0 MMN bkgrd process
3 I 0 1 0 0.00 0.00 0.00 0 Deferred Events
163 I 0 1 0 0.00 0.00 0.00 0 TRACK Main thread
164 I 0 1 0 0.00 0.00 0.00 0 TRACK Client thread
165 I 0 1 0 0.00 0.00 0.00 0 VRRP Main thread
166 I 72436 3664499 0 0.00 0.00 0.00 0 VRRS Main thread
93 I 0 1 0 0.00 0.00 0.00 0 GALIOS DOT1X Proces
168 I 0 1 0 0.00 0.00 0.00 0 avc_dns_as_client
169 I 0 1 0 0.00 0.00 0.00 0 L2FIB Timer Dispatc
170 I 0 1 0 0.00 0.00 0.00 0 MLRIB L2 Msg Thread
171 I 0 1 0 0.00 0.00 0.00 0 sw_vlan_nova IIF Er
172 I 3892 239638 0 0.00 0.00 0.00 0 mDNS
173 I 356 23622 0 0.00 0.00 0.00 0 Ethchnl
174 I 0 2 0 0.00 0.00 0.00 0 MKA
175 I 8 22 0 0.00 0.00 0.00 0 AAA Server
176 I 4 20 0 0.00 0.00 0.00 0 AAA ACCT Proc
177 I 0 1 0 0.00 0.00 0.00 0 ACCT Periodic Proc
178 I 0 1 0 0.00 0.00 0.00 0 Webauth client
179 I 0 1 0 0.00 0.00 0.00 0 Auth-proxy HTTP dae
98 I 0 9 0 0.00 0.00 0.00 0 CEF RRP RF waiter
181 I 92320 326524 0 0.00 0.00 0.00 0 CDP Protocol
182 I 0 1 0 0.00 0.00 0.00 0 HA-IDB-SYNC
183 I 0 2 0 0.00 0.00 0.00 0 ipc thread
184 I 0 2 0 0.00 0.00 0.00 0 AAA Dictionary Recy
185 I 44 1969 0 0.00 0.00 0.00 0 DHCP Snooping
186 I 0 1 0 0.00 0.00 0.00 0 DHCP Snooping db
187 I 0 1 0 0.00 0.00 0.00 0 HQF TARGET DYNAMIC
188 I 32 44 0 0.00 0.00 0.00 0 IP ARP Adjacency
103 I 0 7 0 0.00 0.00 0.00 0 MSP HA IPC flow con
2 I 8432 47249 0 0.00 0.00 0.00 0 Load Meter
191 I 0 1 0 0.00 0.00 0.00 0 ICMP event handler
192 I 4564 459673 0 0.00 0.00 0.00 0 IP ARP Track
193 I 16 6 0 0.00 0.00 0.00 0 PIM register asynch
194 I 0 3 0 0.00 0.00 0.00 0 CEF switching backg
195 I 0 1 0 0.00 0.00 0.00 0 ADJ NSF process
196 I 0 1 0 0.00 0.00 0.00 0 IPv6 ping process
197 I 0 3 0 0.00 0.00 0.00 0 OBFL Cfg Dispatcher
198 I 0 2 0 0.00 0.00 0.00 0 RIB LM VALIDATE
106 I 0 2 0 0.00 0.00 0.00 0 avc_dns_as_ha
200 I 4 14 0 0.00 0.00 0.00 0 SpanTree Msg
201 I 0 21 0 0.00 0.00 0.00 0 Auth Manager
202 I 0 1 0 0.00 0.00 0.00 0 XDR background proc
203 I 752 8471 0 0.00 0.00 0.00 0 XDR mcast
204 I 0 3 0 0.00 0.00 0.00 0 XDR RP Ping Backgro
205 I 5392 23253 0 0.00 0.00 0.00 0 XDR receive
113 I 0 1 0 0.00 0.00 0.00 0 Crypto PKI-HA
207 I 0 1 0 0.00 0.00 0.00 0 XDR RP Test Backgro
208 I 36 3938 0 0.00 0.00 0.00 0 FRR Background Proc
209 I 216 5941 0 0.00 0.00 0.00 0 CEF background proc
210 I 0 1 0 0.00 0.00 0.00 0 fib_fib_bfd_sb even
211 I 0 1 0 0.00 0.00 0.00 0 IP IRDP
212 I 0 1 0 0.00 0.00 0.00 0 Tunnel FIB
213 I 2260 133264 0 0.00 0.00 0.00 0 CEF: IPv4 process
214 I 42964 304043 0 0.00 0.00 0.00 0 ADJ background
215 I 0 10 0 0.00 0.00 0.00 0 Collection process
216 I 0 2 0 0.00 0.00 0.00 0 Routing Topology Pr
217 I 52 38 0 0.00 0.00 0.00 0 IP RIB Update
218 I 68 4064 0 0.00 0.00 0.00 0 IP Background
141 I 13288 1586680 0 0.00 0.00 0.00 0 Ethernet Timer CFM
220 I 0 1 0 0.00 0.00 0.00 0 CFMPAL Process
221 I 0 1 0 0.00 0.00 0.00 0 source guard
222 I 448 3362 0 0.00 0.00 0.00 0 TCP Timer
223 I 0 18 0 0.00 0.00 0.00 0 TCP Protocols
224 I 0 1 0 0.00 0.00 0.00 0 Socket Timers
225 I 24 810 0 0.00 0.00 0.00 0 HTTP CORE
226 I 0 1 0 0.00 0.00 0.00 0 EPM MAIN PROCESS
227 I 6040 469949 0 0.00 0.00 0.00 0 SVM UT Process
228 I 0 1 0 0.00 0.00 0.00 0 CMD HANDLER
229 I 0 1 0 0.00 0.00 0.00 0 DSensor HTTP
230 I 0 4 0 0.00 0.00 0.00 0 DSENSOR HA
231 I 0 1 0 0.00 0.00 0.00 0 COPS
232 I 0 7 0 0.00 0.00 0.00 0 ECFM HA IPC flow co
233 I 0 1 0 0.00 0.00 0.00 0 Probe Input
234 I 88 1752 0 0.00 0.00 0.00 0 MFIB Master backgro
235 I 2184 16772 0 0.00 0.00 0.00 0 MRIB RP PROCESS IPC
236 I 2228 16205 0 0.00 0.00 0.00 0 MRIB RP Proxy
237 I 8 2 0 0.00 0.00 0.00 0 IP SNMPV6
238 I 80 3997 0 0.00 0.00 0.00 0 Call Home Timer Pro
239 I 0 1 0 0.00 0.00 0.00 0 RARP Input
240 I 8 70 0 0.00 0.00 0.00 0 CEF RP Background
241 I 0 4 0 0.00 0.00 0.00 0 XDR FOF process
242 I 0 2 0 0.00 0.00 0.00 0 REP LSL Proc
243 I 0 2 0 0.00 0.00 0.00 0 REP BPA/EPA Proc
244 I 0 1 0 0.00 0.00 0.00 0 Nova UDP
245 I 0 1 0 0.00 0.00 0.00 0 Critical Auth
246 I 0 2 0 0.00 0.00 0.00 0 Dot1x Supplicant Pr
247 I 0 2 0 0.00 0.00 0.00 0 Dot1x Supplicant Ba
248 I 0 2 0 0.00 0.00 0.00 0 Dot1x Supplicant DH
249 I 0 1 0 0.00 0.00 0.00 0 MAB Framework
250 I 0 4 0 0.00 0.00 0.00 0 AAA HA
251 I 4 1 0 0.00 0.00 0.00 0 AAA HA cleanup
252 I 8 138 0 0.00 0.00 0.00 0 IGMPSN L2MCM
253 I 2668 46450 0 0.00 0.00 0.00 0 IGMPSN MRD
254 I 2260 35734 0 0.00 0.00 0.00 0 IGMPSN
255 I 0 1 0 0.00 0.00 0.00 0 IGMPQR
256 I 0 4 0 0.00 0.00 0.00 0 IGMPSN-HA
257 I 0 34 0 0.00 0.00 0.00 0 MLDSN L2MCM
258 I 0 1 0 0.00 0.00 0.00 0 MRD
259 I 0 1 0 0.00 0.00 0.00 0 MLD_SNOOP
260 I 0 4 0 0.00 0.00 0.00 0 MLDSN-HA
261 I 0 4 0 0.00 0.00 0.00 0 MVR-HA
262 I 0 2 0 0.00 0.00 0.00 0 AUTO LAG Protocol
263 I 0 3 0 0.00 0.00 0.00 0 DHCP Snooping HA
264 I 0 43 0 0.00 0.00 0.00 0 Flow Exporter Timer
265 I 0 2 0 0.00 0.00 0.00 0 Tunnel
266 I 0 2 0 0.00 0.00 0.00 0 Media Services Prox
267 I 0 2 0 0.00 0.00 0.00 0 App Route Process
268 I 0 1 0 0.00 0.00 0.00 0 IPv6 RIB Cleanup Ha
269 I 0 3 0 0.00 0.00 0.00 0 IPv6 RIB Event Hand
270 I 4 2 0 0.00 0.00 0.00 0 L2FIB Event Dispatc
271 I 0 6 0 0.00 0.00 0.00 0 SNMP Timers
272 I 12 6 0 0.00 0.00 0.00 0 os_info_p provider
273 I 36 55 0 0.00 0.00 0.00 0 snmp_spi Process
1 I 24 240 0 0.00 0.00 0.00 0 Chunk Manager
138 I 0 1 0 0.00 0.00 0.00 0 EAP Framework
162 I 6492 235796 0 0.00 0.00 0.00 0 FHRP Main thread
277 I 376 23827 0 0.00 0.00 0.00 0 FFM OG-ACL Manager
278 I 0 1 0 0.00 0.00 0.00 0 SNMP ConfCopyProc
279 I 0 1 0 0.00 0.00 0.00 0 Nova TCP
280 I 0 1 0 0.00 0.00 0.00 0 Nova RAW
281 I 0 1 0 0.00 0.00 0.00 0 Port Range Departur
282 I 196 139 0 0.00 0.00 0.00 0 SNMP Traps
283 I 0 4 0 0.00 0.00 0.00 0 BFD HA
284 I 4 3 0 0.00 0.00 0.00 0 LIM SPI
285 I 0 1 0 0.00 0.00 0.00 0 AAA System Acct Pro
286 I 0 1 0 0.00 0.00 0.00 0 EPM Downlad Manager
287 I 0 1 0 0.00 0.00 0.00 0 Timer Handler Proce
288 I 0 1 0 0.00 0.00 0.00 0 HTTP EPM Redirect D
289 I 1752 47892 0 0.00 0.00 0.00 0 EPM ACL PLUG-IN PRO
290 I 8 25 0 0.00 0.00 0.00 0 KEYSTORE HA IPC PRO
291 I 0 1 0 0.00 0.00 0.00 0 DCT Bkgd Process
292 I 0 5 0 0.00 0.00 0.00 0 HSRP HA
293 I 0 3 0 0.00 0.00 0.00 0 Service Routing MCP
294 I 4 42 0 0.00 0.00 0.00 0 SR CapMan Process
295 I 68 48 0 0.00 0.00 0.00 0 MVPN Mgr Process
296 I 0 2 0 0.00 0.00 0.00 0 Multicast Offload
297 I 8 19 0 0.00 0.00 0.00 0 ADJ resolve process
298 I 0 21 0 0.00 0.00 0.00 0 static
299 I 0 1 0 0.00 0.00 0.00 0 IPv6 Static Handler
300 I 26268 25224 0 0.00 0.00 0.00 0 QoS stats process
301 I 71724 3664515 0 0.00 0.00 0.00 0 MMA DB TIMER
302 I 0 1 0 0.00 0.00 0.00 0 MMA MSG MCOL
303 I 0 1 0 0.00 0.00 0.00 0 Opaque Database
304 I 0 1 0 0.00 0.00 0.00 0 mDNS snooping
305 I 0 2 0 0.00 0.00 0.00 0 DHCP IOSd Agent
306 I 0 1 0 0.00 0.00 0.00 0 IPSECv6 PS Proc
307 I 0 2 0 0.00 0.00 0.00 0 AAA Cached Server G
308 I 0 1 0 0.00 0.00 0.00 0 EM Background Proce
309 I 0 1 0 0.00 0.00 0.00 0 Key chain livekeys
310 I 0 3 0 0.00 0.00 0.00 0 LOCAL AAA
311 I 0 2 0 0.00 0.00 0.00 0 ENABLE AAA
312 I 0 2 0 0.00 0.00 0.00 0 LINE AAA
313 I 0 3 0 0.00 0.00 0.00 0 LDAP process
314 I 128 355 0 0.00 0.00 0.00 0 OBFL UPTIME slot-11
315 I 0 2 0 0.00 0.00 0.00 0 TPLUS
316 I 8 334 0 0.00 0.00 0.00 0 NIST rng proc
317 I 4760 459673 0 0.00 0.00 0.00 0 Timer Library
318 I 0 2 0 0.00 0.00 0.00 0 Crypto Support
319 I 0 12 0 0.00 0.00 0.00 0 PIM HA
320 I 0 2 0 0.00 0.00 0.00 0 REP Switch Helper P
321 I 0 2 0 0.00 0.00 0.00 0 MMON PROCESS
322 I 70036 3664522 0 0.00 0.00 0.00 0 MMA DP TIMER
180 I 0 1 0 0.00 0.00 0.00 0 IP Admin SM Process
324 I 20 13 0 0.00 0.00 0.00 0 crypto engine proc
325 I 0 1 0 0.00 0.00 0.00 0 RSA background proc
326 I 56 266 0 0.00 0.00 0.00 0 Crypto CA
327 I 0 3 0 0.00 0.00 0.00 0 Crypto PKI-CRL
328 I 0 1 0 0.00 0.00 0.00 0 Crypto PKI-Rev-Peer
329 I 0 1 0 0.00 0.00 0.00 0 PKI OCSP
330 I 0 1 0 0.00 0.00 0.00 0 PKI Revocation
331 I 0 1 0 0.00 0.00 0.00 0 encrypt proc
332 I 0 3 0 0.00 0.00 0.00 0 CRYPTO MAP FREE DEF
333 I 0 1 0 0.00 0.00 0.00 0 Crypto INT
334 I 0 2 0 0.00 0.00 0.00 0 Crypto IKE Dispatch
335 I 0 3 0 0.00 0.00 0.00 0 Crypto IKMP
336 I 0 1 0 0.00 0.00 0.00 0 IPSEC manual key
337 I 140 11813 0 0.00 0.00 0.00 0 IPSEC key engine
338 I 0 1 0 0.00 0.00 0.00 0 Crypto ACL
339 I 0 1 0 0.00 0.00 0.00 0 Crypto PAS Proc
340 I 0 1 0 0.00 0.00 0.00 0 IPSec background pr
341 I 7676 336007 0 0.00 0.00 0.00 0 Crypto IKEv2
342 I 0 1 0 0.00 0.00 0.00 0 IKEv2 AAA handler
343 I 748 8136 0 0.00 0.00 0.00 0 Galios Fnf
344 I 0 5 0 0.00 0.00 0.00 0 OBFL ENV slot-11
345 I 0 2 0 0.00 0.00 0.00 0 DHCP Security Helpe
346 I 320 8041 0 0.00 0.00 0.00 0 DiagCard1/-1
347 I 336 8086 0 0.00 0.00 0.00 0 DiagCard11/-1
348 I 11588 631986 0 0.00 0.00 0.00 0 PM Callback
349 I 300 19260 0 0.00 0.00 0.00 0 dhcp snooping sw pr
350 I 0 12 0 0.00 0.00 0.00 0 VTP Trap Process
351 I 18704 123784 0 0.00 0.00 0.00 0 Stats Sync
352 I 0 4 0 0.00 0.00 0.00 0 EEM ED Media Servic
353 I 0 1 0 0.00 0.00 0.00 0 Licensing MIB proce
354 I 0 1 0 0.00 0.00 0.00 0 Vlan_port_nova IIF
355 I 0 17 0 0.00 0.00 0.00 0 OBFL MSG slot-11
356 I 0 29 0 0.00 0.00 0.00 0 ASP Process Creatio
357 I 0 1 0 0.00 0.00 0.00 0 OBFL INTR slot-11
358 I 0 1 0 0.00 0.00 0.00 0 FFCP msg process
206 I 0 4 0 0.00 0.00 0.00 0 IPC LC Message Hand
360 I 0 20 0 0.00 0.00 0.00 0 AAA SEND STOP EVENT
361 I 0 1 0 0.00 0.00 0.00 0 Test AAA Client
362 I 4 1 0 0.00 0.00 0.00 0 dcm_cli_engine
363 I 0 3 0 0.00 0.00 0.00 0 dcm_cli_provider
364 I 0 5 0 0.00 0.00 0.00 0 DCM Core Thread
365 I 36 1118 0 0.00 0.00 0.00 0 EEM ED Syslog
366 I 0 4 0 0.00 0.00 0.00 0 EEM ED Routing
367 I 0 4 0 0.00 0.00 0.00 0 EEM ED Generic
368 I 0 3 0 0.00 0.00 0.00 0 Online Diag EEM Tas
369 I 12 6 0 0.00 0.00 0.00 0 Syslog Traps
370 I 4 3 0 0.00 0.00 0.00 0 dcm_expression_prov
371 I 560 24088 0 0.00 0.00 0.00 0 Bulkstat-Client
372 I 0 3 0 0.00 0.00 0.00 0 SPAN switch
373 I 0 1 0 0.00 0.00 0.00 0 IOSD-ISSU commitver
374 I 0 1 0 0.00 0.00 0.00 0 IOSXE Virtual servi
375 I 0 1 0 0.00 0.00 0.00 0 IP SLAs Ethernet Ev
376 I 4 77 0 0.00 0.00 0.00 0 EEM Server
377 I 0 3 0 0.00 0.00 0.00 0 EEM Policy Director
378 I 8 26 0 0.00 0.00 0.00 0 RBM CORE
379 I 0 3 0 0.00 0.00 0.00 0 CTS credentials pro
380 I 476 2006 0 0.00 0.00 0.00 0 Syslog
381 I 4 3 0 0.00 0.00 0.00 0 Call Home process
382 I 24 64 0 0.00 0.00 0.00 0 RADIUS
383 I 0 4 0 0.00 0.00 0.00 0 EEM ED CLI
384 I 0 4 0 0.00 0.00 0.00 0 EEM ED Counter
385 I 0 4 0 0.00 0.00 0.00 0 EEM ED Interface
386 I 0 4 0 0.00 0.00 0.00 0 EEM ED IOSWD
387 I 0 4 0 0.00 0.00 0.00 0 EEM ED None
388 I 0 4 0 0.00 0.00 0.00 0 EEM ED OIR
389 I 0 17 0 0.00 0.00 0.00 0 EEM ED RF
390 I 0 4 0 0.00 0.00 0.00 0 EEM ED SNMP
391 I 152 4409 0 0.00 0.00 0.00 0 EEM ED Timer
392 I 0 4 0 0.00 0.00 0.00 0 EEM ED SNMP Object
393 I 4 4 0 0.00 0.00 0.00 0 EEM ED SNMP Noti
394 I 0 4 0 0.00 0.00 0.00 0 EEM ED Ipsla
395 I 0 4 0 0.00 0.00 0.00 0 EEM ED Nf
396 I 0 4 0 0.00 0.00 0.00 0 EEM ED Test
397 I 0 4 0 0.00 0.00 0.00 0 EEM ED Config
398 I 0 4 0 0.00 0.00 0.00 0 EEM ED Env
399 I 0 4 0 0.00 0.00 0.00 0 EEM ED DS
400 I 0 4 0 0.00 0.00 0.00 0 EEM ED CRASH
401 I 0 4 0 0.00 0.00 0.00 0 EM ED GOLD
402 I 0 4 0 0.00 0.00 0.00 0 EEM ED RPC
403 I 0 1 0 0.00 0.00 0.00 0 PnP-Monitor
404 I 0 3 0 0.00 0.00 0.00 0 MFIB PROCESS IPC MS
405 I 20 12 0 0.00 0.00 0.00 0 VLAN Manager
406 I 3020 14826 0 0.00 0.00 0.00 0 mdns Timer Process
407 I 0 2 0 0.00 0.00 0.00 0 MRIB Process
408 I 0 13 0 0.00 0.00 0.00 0 EEM Helper Thread
409 I 0 4 0 0.00 0.00 0.00 0 L2FIB HA Flow Threa
410 I 24 1969 0 0.00 0.00 0.00 0 Secure Login
411 I 65544 2496826 0 0.00 0.00 0.00 0 IGMP Input
219 I 12 177 0 0.00 0.00 0.00 0 IP Connected Route
413 I 37268 2410391 0 0.00 0.00 0.00 0 Mwheel Process
414 I 296 8809 0 0.00 0.00 0.00 0 mcast_rw_process
415 I 1912 51057 0 0.00 0.00 0.00 0 MRIB Trans
416 I 0 4 0 0.00 0.00 0.00 0 MFIB_init
417 I 4 34 0 0.00 0.00 0.00 0 MFIB_backwalk
418 I 0 2 0 0.00 0.00 0.00 0 MFIB_ppr_release
419 I 20 16 0 0.00 0.00 0.00 0 MFIB_cable
420 I 141264 1057972 0 0.00 0.00 0.00 0 MFIB_mrib_write
421 I 9932 19654 0 0.00 0.00 0.00 0 MFIB_mrib_read
422 I 496 29524 0 0.00 0.00 0.00 0 MFIB_rate
423 I 1844 59612 0 0.00 0.00 0.00 0 MFIB_hw_stats
424 I 3368 236235 0 0.00 0.00 0.00 0 FNF Cache Ager Proc
425 I 26556 379510 0 0.00 0.00 0.00 0 LACP Protocol
426 I 396 5882 0 0.00 0.00 0.00 0 OBFL TEMP slot-11
427 I 0 1 0 0.00 0.00 0.00 0 IPv6 Access Control
428 I 12788 246645 0 0.00 0.00 0.00 0 NTP
429 I 4 6 0 0.00 0.00 0.00 0 DCM snmp dp Thread
430 I 16 50 0 0.00 0.00 0.00 0 SpanTree Helper
431 I 140 3937 0 0.00 0.00 0.00 0 RADIUS IO STATS
432 I 13500 483858 0 0.00 0.00 0.00 0 OSPF-10 Router
433 I 17724 280601 0 0.00 0.00 0.00 0 OSPF-10 Hello
434 I 0 3 0 0.00 0.00 0.00 0 snmp dcm ma shim th
435 I 0 1 0 0.00 0.00 0.00 0 License Client Lib
436 I 2424 235532 0 0.00 0.00 0.00 0 Inline power increm
437 I 0 1 0 0.00 0.00 0.00 0 License Restart wat

------------------------------------------------------------------------------------------------------------------

SWITCH_4500X#show ip traffic
IP statistics:
Rcvd: 1044803460 total, 518090905 local destination
0 format errors, 0 checksum errors, 11760 bad hop count
1 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 70835 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 70835 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
0 invalid hole
Bcast: 509211533 received, 0 sent
Mcast: 133118 received, 181385 sent
Sent: 8926588 generated, 410098396 forwarded
Drop: 1066 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
3386 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 0 unreachable, 0 echo, 3386 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 19 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

TCP statistics:
Rcvd: 5764 total, 0 checksum errors, 10 no port
Sent: 6571 total

Probe statistics:
Rcvd: 0 address requests, 0 address replies
0 proxy name requests, 0 where-is requests, 0 other
Sent: 0 address requests, 0 address replies (0 proxy)
0 proxy name replies, 0 where-is replies

PIMv2 statistics: Sent/Received
Total: 93404/64808, 0 checksum errors, 0 format errors
Registers: 1949/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/1949, Hellos: 64018/39387
Join/Prunes: 27495/23472, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 35442/19704, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 31503/3940, Host Reports: 3939/15760, Host Leaves: 0/4
DVMRP: 0/0, PIM: 0/0
Queue drops: 0
Report VRF mismatch drops: 0

UDP statistics:
Rcvd: 517946600 total, 0 checksum errors, 509211551 no port 0 finput
Sent: 8736313 total, 0 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 50642 total, 0 checksum errors
48921 hello, 21 database desc, 5 link state req
598 link state updates, 1096 link state acks
Sent: 51452 total
49720 hello, 18 database desc, 2 link state req
1348 link state updates, 364 link state acks

ARP statistics:
Rcvd: 310 requests, 180 replies, 0 reverse, 0 other
Sent: 507 requests, 273 replies (0 proxy), 0 reverse
Drop due to input queue full: 0

-------------------------------------------------------------------------------------------------------------

SWITCH_4500X#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
A.B.21.142 1 FULL/BDR 00:00:39 A.B.5.42 Vlan189
A.B.21.138 1 FULL/DR 00:00:34 A.B.5.34 Vlan188

---------------------------------------------------------------------------------------------------------------

SWITCH_4500X#show ip ospf interface
Loopback0 is up, line protocol is up
Internet Address A.B.5.130/32, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type LOOPBACK, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Loopback interface is treated as a stub Host
Vlan3000 is up, line protocol is up
Internet Address A.B.40.1/29, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) A.B.5.130, Interface address A.B.40.1
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
No Hellos (Passive interface)
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/9/9, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)
Cryptographic authentication enabled
No key configured, using default key id 0
TenGigabitEthernet2/1/12 is administratively down, line protocol is down (disabled)
Internet Address A.B.5.57/30, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DOWN, Priority 255
No designated router on this network
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
No Hellos (Passive interface)
TenGigabitEthernet1/1/12 is up, line protocol is up (connected)
Internet Address A.B.5.53/30, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) A.B.5.130, Interface address A.B.5.53
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
No Hellos (Passive interface)
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/6/6, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)
Cryptographic authentication enabled
No key configured, using default key id 0
Vlan189 is up, line protocol is up
Internet Address A.B.5.41/29, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) A.B.5.130, Interface address A.B.5.41
Backup Designated router (ID) A.B.21.142, Interface address A.B.5.42
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:07
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/5/5, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 1, maximum is 9
Last flood scan time is 0 msec, maximum is 4 msec
Neighbor Count is 1, Adjacent neighbor count is 1
Adjacent with neighbor A.B.21.142 (Backup Designated Router)
Suppress hello for 0 neighbor(s)
Cryptographic authentication enabled
Youngest key id is 1
Vlan188 is up, line protocol is up
Internet Address A.B.5.33/29, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State BDR, Priority 1
Designated Router (ID) A.B.21.138, Interface address A.B.5.34
Backup Designated router (ID) A.B.5.130, Interface address A.B.5.33
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
Hello due in 00:00:04
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/4/4, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 2, maximum is 11
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 1, Adjacent neighbor count is 1
Adjacent with neighbor A.B.21.138 (Designated Router)
Suppress hello for 0 neighbor(s)
Cryptographic authentication enabled
Youngest key id is 1
Vlan356 is up, line protocol is up
Internet Address A.B.5.17/30, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) A.B.5.130, Interface address A.B.5.17
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
No Hellos (Passive interface)
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/3/3, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)
Cryptographic authentication enabled
Youngest key id is 1
Vlan167 is up, line protocol is up
Internet Address A.B.5.9/30, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) A.B.5.130, Interface address A.B.5.9
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
No Hellos (Passive interface)
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/2/2, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)
Cryptographic authentication enabled
No key configured, using default key id 0
Vlan1633 is up, line protocol is up
Internet Address A.B.5.1/30, Area 0, Attached via Network Statement
Process ID 10, Router ID A.B.5.130, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) A.B.5.130, Interface address A.B.5.1
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
No Hellos (Passive interface)
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/1/1, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)
Cryptographic authentication enabled
No key configured, using default key id 0

-----------------------------------------------------------------------------------------------

SWITCH_4500X#show ip ospf database database-summary

OSPF Router with ID (A.B.5.130) (Process ID 10)

Area 0 database summary
LSA Type Count Delete Maxage
Router 3 0 0
Network 2 0 0
Summary Net 0 0 0
Summary ASBR 0 0 0
Type-7 Ext 0 0 0
Prefixes redistributed in Type-7 0
Opaque Link 0 0 0
Opaque Area 0 0 0
Subtotal 5 0 0

Process 10 database summary
LSA Type Count Delete Maxage
Router 3 0 0
Network 2 0 0
Summary Net 0 0 0
Summary ASBR 0 0 0
Type-7 Ext 0 0 0
Opaque Link 0 0 0
Opaque Area 0 0 0
Type-5 Ext 14 0 0
Prefixes redistributed in Type-5 11
Opaque AS 0 0 0
Non-self 6
Total 19 0 0

KGrev · ‎08-21-2023

@Peter PaluchSorry theres nothing in the logs currently. As stated in the original post I did do a reload that cleared the logs. Also the ospf failures subsided after that. But I've done a reload for this before and eventually it comes back. Also the latency in pings to the loopback are still present.

Peter Paluch · ‎08-21-2023

Hello @KGrev ,

Thank you very much for sharing the outputs. Before I dive deeper, one humble request: In the future, if possible, save the outputs in a file and attach the file to your post, instead of posting the output directly. When you just copy&paste the outputs, aside from them being impractically long, their formatting is lost and they are very difficult to read.

As for the CPU load, it does not appear to be abnormal (yet), and the processes consuming the most CPU time are fairly common to see there - so that does not give us a clue.

Looking at the "show ip traffic" output, however, I am surprised to see quite a large amount of received broadcasts. Do you know what application in your network could be responsible for generating a large amount of IP broadcasts? It might be that the amount of IP broadcasts is excessive and gets in the way of processing OSPF packets or ICMP pings. Hence, I would like to narrow down the cause of those IP broadcasts before we move on.

We have an option of recording the traffic that hits your supervisor into a PCAP file and have a look at it in Wireshark. With switches, we should be seeing very little traffic on the supervisor because all flows should be handled in hardware - so any extra traffic there is a potential reason to troubleshoot. This PCAP file would be very helpful but at the same time, it cannot be sanitized the way text outputs are sanitized, so it may hold sensitive information - it is up to you to decide if you first want to even try to do this control plane packet capture, and if so, whether you would like to analyze the file in Wireshark yourself, or whether you can also share it here.

Best regards,
Peter

KGrev · ‎08-25-2023

Marked Peters answer as the answer. Conducting captures on many of the interfaces showed a broadcast flood from a device. After resolving that things seem normal. Unsure if that was the cause but will continue to monitor.

Peter Paluch · ‎08-26-2023

Hello,

I am glad to hear that!

One suggestion: It seems you performed a per-port traffic capture. That surely worked but was also quite tedious to do. Because we have been primarily interested in the traffic that is reaching the supervisor from any port of the switch, one way of doing it was to perform a monitor capture on the control plane itself. This can be done as follows:

monitor capture alltraf buffer size 50
monitor capture alltraf match any
monitor capture alltraf control-plane both
monitor capture alltraf start

This will set up a packet capture of all traffic that is received or sent by the CPU of the switch (note - it's not all traffic that traverses the switch, only the traffic that is punted to the CPU by the forwarding hardware or originated by the operating system - which is exactly what we're looking into in these cases). The buffer size to hold the captured traffic is set to 50 MB - after this size, the capture will stop automatically.

After executing these commands, wait for a few minutes (up to you how long), and then stop the capture and export the capture traffic as follows:

monitor capture alltraf stop
monitor capture alltraf export bootflash:traffic.pcap

You can then download the file from the bootflash: filesystem to your computer and analyze it with Wireshark. If you find any suspicious traffic, you can trace it back based on the source IP addresses and MAC addresses - but I think I'm now preaching to the choir...

Maybe this proves useful at some point in future.

Best regards,
Peter

KGrev · ‎08-31-2023

Another update: The issue came back so we were off the first time but using @Peter Paluch advice for control-plane capture we were able to see an individual spamming broadcast. We tracked it down to their port and monitored traffic with their port disabled.

Clear "show ip traffic" counters
---------------(1 minute capture)----------------

IP statistics:
Rcvd: 591074 total, 293825 local destination
0 format errors, 0 checksum errors, 3 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 20 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 20 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
0 invalid hole
Bcast: 290247 received, 0 sent<------------------------------------------
Mcast: 38 received, 50 sent
Sent: 3587 generated, 174764 forwarded
Drop: 0 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero

Clear "show ip traffic" counters
---------------(1 minute capture with port shut)----------------

IP statistics:
Rcvd: 10056 total, 3403 local destination
0 format errors, 0 checksum errors, 3 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 21 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 21 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
0 invalid hole
Bcast: 0 received, 0 sent <------------------------------------------
Mcast: 44 received, 64 sent
Sent: 3423 generated, 253426 forwarded
Drop: 0 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero

MHM Cisco World · ‎08-31-2023

I am so busy' if I dont reply forgive me.

I see you select solution but I was not sure it issue here

But the SW receive alot of IP' the SW should not receive it.

The SW send traffic to CPU instead of bridging it in case that packet have option or there is fragment (the fragment count is zero so this not case here) and etc.

So share config I will make check' but again if I dont answer forgive me. I am so busy until 12/9' hope answer you before that.

Thanks

MHM

KGrev · ‎08-31-2023

@MHM Cisco WorldThank you for your response. I was just giving more context in case someone stumbled onto this thread. My issue has a target now I can work with. Just wanted to give someone more to troubleshoot with if they ran into a similar issue.