本帖最后由 iosvip@163.com 于 2020-6-11 11:03 编辑 6509交换机HSRP状态不定期切换,不知是否与CPU利用率过高有关系。请各位大咖支持,谢谢!涉及可能的原因有哪些?排查思路?
基础信息:
设备版本:s72033-adventerprisek9_wan-mz.122-33.SXJ5组网拓扑:
7609(主)---------6509(备) \ / \ / 思科其他29/35等交换机
主要配置:7609(主)interface Vlan2
description JuJi** vlan
ip address 10.x.97.200 255.255.255.0
ip helper-address 10.x.96.5
ip helper-address 10.x.96.6
no ip redirects
no ip unreachables
ip access-group vlan_acl out
standby 2 ip 10.x.97.254
standby 2 priority 130
standby 2 preempt
!
interface Vlan3
description Oems*
ip address 10.232.98.126 255.255.255.128
ip helper-address 10.x.96.5
ip helper-address 10.x.96.6
no ip redirects
no ip unreachables
standby 2 ip 10.x.98.62
standby 2 priority 130
standby 2 preempt
!
interface Vlan99
description "Cai***"
ip address 10.232.101.155 255.255.255.224
ip helper-address 10.x.96.5
ip helper-address 10.x.96.6
no ip redirects
ip access-group vlan_acl out
standby 2 ip 10.x.101.129
standby 2 priority 130
standby 2 preempt
6509(备)interface Vlan2
description juji* vlan
ip address 10.x.97.253 255.255.255.0
ip helper-address 10.x.96.5
ip helper-address 10.x.96.6
no ip redirects
no ip unreachables
ip flow ingress
standby 2 ip 10.x.97.254
standby 2 priority 120
standby 2 preempt
no ip mroute-cache
!
interface Vlan3
description oems*
ip address 10.x.98.61 255.255.255.128
ip helper-address 10.x.96.5
ip helper-address 10.x.96.6
no ip redirects
no ip unreachables
standby 2 ip 10.x.98.62
standby 2 priority 120
standby 2 preempt
no ip mroute-cache
interface Vlan99
description "ca***"
ip address 10.x.101.131 255.255.255.224
ip access-group vlan_acl out
ip helper-address 10.x.96.5
ip helper-address 10.x.96.6
standby 2 ip 10.x.101.129
standby 2 priority 120
standby 2 preempt
如下为:5月15日,在6509交换机上HSRP状态切换日志,在7609上未用切换日志信息。
据用户描述,随后查看CPU利用率在70%以上,主要涉及IP Input 和 arp Input 两个进程。(持续时间短,3分钟左右)
当时的CPU利用率未做记录,文后有采集到其他时间CPU利用率过高的记录和相应日志信息。
C6509,5.15号日志信息:
May 15 07:24:07: %HSRP-5-STATECHANGE: Vlan202 Grp 2 state Standby -> Active
May 15 07:24:08: %HSRP-5-STATECHANGE: Vlan202 Grp 2 state Active -> Speak
May 15 07:24:18: %HSRP-5-STATECHANGE: Vlan202 Grp 2 state Speak -> Standby
May 15 07:24:20: %HSRP-5-STATECHANGE: Vlan5 Grp 5 state Standby -> Active
May 15 07:24:21: %HSRP-5-STATECHANGE: Vlan6 Grp 2 state Standby -> Active
May 15 07:24:21: %HSRP-5-STATECHANGE: Vlan6 Grp 2 state Active -> Speak
May 15 07:24:25: %HSRP-5-STATECHANGE: Vlan5 Grp 5 state Active -> Speak
May 15 07:24:32: %HSRP-5-STATECHANGE: Vlan6 Grp 2 state Speak -> Standby
May 15 07:24:36: %HSRP-5-STATECHANGE: Vlan5 Grp 5 state Speak -> Standby
May 15 07:24:42: %HSRP-5-STATECHANGE: Vlan5 Grp 5 state Standby -> Active
May 15 07:24:42: %HSRP-5-STATECHANGE: Vlan5 Grp 5 state Active -> Speak
May 15 07:24:42: %HSRP-5-STATECHANGE: Vlan6 Grp 2 state Standby -> Active
May 15 07:24:42: %HSRP-5-STATECHANGE: Vlan7 Grp 7 state Standby -> Active
May 15 07:24:42: %HSRP-5-STATECHANGE: Vlan7 Grp 7 state Active -> Speak
May 15 07:24:44: %HSRP-5-STATECHANGE: Vlan201 Grp 2 state Standby -> Active
....
May 15 07:55:24: %HSRP-5-STATECHANGE: Vlan5 Grp 5 state Active -> Speak
May 15 07:55:24: %HSRP-5-STATECHANGE: Vlan201 Grp 2 state Active -> Speak
May 15 07:55:25: %HSRP-5-STATECHANGE: Vlan99 Grp 2 state Speak -> Standby
May 15 07:55:26: %HSRP-5-STATECHANGE: Vlan99 Grp 2 state Standby -> Active
May 15 07:55:26: %HSRP-5-STATECHANGE: Vlan99 Grp 2 state Active -> Speak
May 15 07:55:26: %HSRP-5-STATECHANGE: Vlan22 Grp 5 state Standby -> Active
May 15 07:55:26: %HSRP-5-STATECHANGE: Vlan2 Grp 2 state Speak -> Standby
May 15 07:55:29: %HSRP-5-STATECHANGE: Vlan30 Grp 30 state Standby -> Active
May 15 07:55:30: %HSRP-5-STATECHANGE: Vlan4 Grp 4 state Speak -> Standby
May 15 07:55:30: %HSRP-5-STATECHANGE: Vlan22 Grp 5 state Active -> Speak
May 15 07:55:31: %HSRP-5-STATECHANGE: Vlan4 Grp 4 state Standby -> Active
May 15 07:55:33: %HSRP-5-STATECHANGE: Vlan202 Grp 2 state Active -> Speak
May 15 07:55:33: %HSRP-5-STATECHANGE: Vlan4 Grp 4 state Active -> Speak
May 15 07:55:34: %HSRP-5-STATECHANGE: Vlan30 Grp 30 state Active -> Speak
May 15 07:55:34: %HSRP-5-STATECHANGE: Vlan201 Grp 2 state Speak -> Standby
May 15 07:55:35: %HSRP-5-STATECHANGE: Vlan5 Grp 5 state Speak -> Standby
May 15 07:55:37: %HSRP-5-STATECHANGE: Vlan99 Grp 2 state Speak -> Standby
May 15 07:55:42: %HSRP-5-STATECHANGE: Vlan22 Grp 5 state Speak -> Standby
May 15 07:55:44: %HSRP-5-STATECHANGE: Vlan4 Grp 4 state Speak -> Standby
May 15 07:55:45: %HSRP-5-STATECHANGE: Vlan202 Grp 2 state Speak -> Standby
May 15 07:55:45: %HSRP-5-STATECHANGE: Vlan30 Grp 30 state Speak -> Standby
7609,5.15号日志信息
May 15 07:33:32.040 BeiJing: %OSPF-5-ADJCHG: Process 1, Nbr 10.232.239.216 on GigabitEthernet4/42 from FULL to DOWN, Neighbor Down: Dead timer expired
May 15 07:33:57.108 BeiJing: %OSPF-5-ADJCHG: Process 1, Nbr 10.232.239.216 on GigabitEthernet4/42 from LOADING to FULL, Loading Done
May 15 07:35:00.983 BeiJing: %OSPF-5-ADJCHG: Process 1, Nbr 100.100.10.2 on GigabitEthernet4/22 from LOADING to FULL, Loading Done
May 15 07:47:18.021 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 4 packets
May 15 07:53:05.264 BeiJing: %OSPF-5-ADJCHG: Process 1, Nbr 10.232.239.216 on GigabitEthernet4/42 from FULL to DOWN, Neighbor Down: Dead timer expired
May 15 07:53:11.296 BeiJing: %OSPF-5-ADJCHG: Process 1, Nbr 100.100.10.2 on GigabitEthernet4/22 from LOADING to FULL, Loading Done
May 15 07:55:34.991 BeiJing: %OSPF-5-ADJCHG: Process 1, Nbr 10.232.239.216 on GigabitEthernet4/42 from LOADING to FULL, Loading Done
May 15 07:57:18.018 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 08:24:42.678 BeiJing: %SYS-5-CONFIG_I: Configured from console by xtepc on vty0 (10.232.106.93)
May 15 08:31:18.011 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 08:42:17.993 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 08:49:17.987 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 09:02:49.907 BeiJing: NTP Core (INFO): 10.232.96.5 961D 8D popcorn popcorn
May 15 09:07:06.899 BeiJing: NTP Core (INFO): 0.0.0.0 0618 08 no_sys_peer
May 15 09:07:06.899 BeiJing: NTP Core (INFO): 0.0.0.0 C61C 0C clock_step
May 15 09:07:06.899 BeiJing: NTP Core (NOTICE): Clock synchronization lost.
.May 15 09:07:17.975 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
.May 15 09:08:10.898 BeiJing: NTP Core (INFO): 0.0.0.0 C02C 0C clock_step
.May 15 09:09:16.897 BeiJing: NTP Core (INFO): 0.0.0.0 C03C 0C clock_step
.May 15 09:10:17.965 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
.May 15 09:10:23.897 BeiJing: NTP Core (INFO): 0.0.0.0 C04C 0C clock_step
.May 15 09:10:23.897 BeiJing: NTP Core (INFO): 10.232.96.5 8023 83 unreachable
.May 15 09:12:17.971 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 4 packets
.May 15 09:17:17.967 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 2 packets
.May 15 09:23:17.958 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
.May 15 09:25:41.888 BeiJing: NTP Core (INFO): 10.232.96.5 8034 84 reachable
.May 15 09:25:41.888 BeiJing: NTP Core (INFO): 0.0.0.0 C05C 0C clock_step
.May 15 09:26:43.883 BeiJing: NTP Core (INFO): 10.232.96.5 964A 8A sys_peer
.May 15 09:26:43.883 BeiJing: NTP Core (NOTICE): Spike event s
.May 15 09:27:46.113 BeiJing: %SYS-6-CLOCKUPDATE: System clock has been updated from 09:27:45 BeiJing Fri May 15 2020 to 09:27:46 BeiJing Fri May 15 2020, configured from NTP by 10.232.96.5.
.May 15 09:27:46.113 BeiJing: NTP Core (NOTICE): time reset 0.000000 s
.May 15 09:27:46.113 BeiJing: NTP Core (NOTICE): trans state : 5
.May 15 09:27:46.113 BeiJing: NTP Core (INFO): 0.0.0.0 C66C 0C clock_step
.May 15 09:27:46.113 BeiJing: NTP Core (INFO): 0.0.0.0 C07C 0C clock_step
.May 15 09:27:47.109 BeiJing: NTP Core (INFO): 10.232.96.5 8054 84 reachable
.May 15 09:27:47.109 BeiJing: NTP Core (INFO): 0.0.0.0 C08C 0C clock_step
.May 15 09:28:18.184 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
.May 15 09:29:50.115 BeiJing: NTP Core (INFO): 10.232.96.5 966A 8A sys_peer
.May 15 09:29:50.115 BeiJing: NTP Core (NOTICE): Clock is synchronized.
May 15 09:34:18.175 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 09:36:18.193 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
May 15 09:42:18.166 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 3 packets
May 15 09:47:18.157 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
May 15 09:50:18.157 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 09:53:18.148 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
May 15 09:58:18.148 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 7 packets
May 15 10:00:18.141 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 10:04:18.136 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 4 packets
May 15 10:09:18.132 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 4 packets
May 15 10:15:18.115 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
May 15 10:21:18.105 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
May 15 10:24:18.105 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.102 2 packets
May 15 10:26:18.102 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 4 packets
May 15 10:32:18.097 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 5 packets
May 15 10:38:18.078 BeiJing: %SEC-6-IPACCESSLOGS: list 99 denied 192.168.200.154 1 packet
May 15 10:46:18.069 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 10:53:18.054 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 11:14:44.950 BeiJing: NTP Core (INFO): 10.232.96.5 967D 8D popcorn popcorn
May 15 11:36:17.993 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 13:44:53.764 BeiJing: NTP Core (INFO): 10.232.96.5 961D 8D popcorn popcorn
May 15 16:56:17.585 BeiJing: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packets
May 15 17:34:01.464 BeiJing: NTP Core (INFO): 10.232.96.5 961D 8D popcorn popcorn
May 15 20:31:06.243 BeiJing: NTP Core (INFO): 10.232.96.5 961D 8D popcorn popcorn
******************************************************************************************************
6月8日的CPU利用率高的情况
后面附6.8日的日志,并没有引起大多数vlan的HSRP状态切换,只vlan99和vlan2,见后面日志信息
6509-720#show process cpu
CPU utilization for fiveseconds: 99%/16%; one minute: 71%; five minutes: 42%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 8 259 30 0.00% 0.00% 0.00% 0 Chunk Manager
2 5208 2624169 1 0.00% 0.00% 0.00% 0 Load Meter
3 160 1928 82 0.00% 0.00% 0.00% 0 Exec
4 0 1 0 0.00% 0.00% 0.00% 0 Connection Mgr
5 0 235 0 0.00% 0.00% 0.00% 0 Retransmission o
6 0 15 0 0.00% 0.00% 0.00% 0 IPC ISSU Dispatc
7 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req
8 0 9 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req
9 68635276 3937650 17430 2.07% 0.54% 0.52% 0 Check heaps
10 3460 3064 1129 0.00% 0.00% 0.00% 0 Pool Manager
11 0 2 0 0.00% 0.00% 0.00% 0 Timers
12 9421248163333766606 282 5.74% 7.24% 8.08% 0 ARP Input
13 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
.....
264 2029832 17903135 113 0.07% 0.13% 0.13% 0 CDP Protocol
265 100 1273 78 0.00% 0.00% 0.00% 0 Collection proce
266 0 23 0 0.00% 0.00% 0.00% 0 MPLS Auto Mesh P
267 0 342 0 0.00% 0.00% 0.00% 0 CDP EXPORT PROCE
268 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
269 0 2 0 0.00% 0.00% 0.00% 0 CEF switching ba
270 5828104602498910166 23358.15% 45.75% 21.02% 0 IP Input
271 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
272 3591280 29849868 120 0.00% 0.04% 0.02% 0 ADJ resolve proc
273 0 1 0 0.00% 0.00% 0.00% 0 PPCP RP Stats Ba
274 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Echo event
275 10688 70290 152 0.00% 0.00% 0.00% 0 MOP Protocols
6月8日,日志信息:
Jun 8 10:03:34: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.102 2 packetsJun 8 10:08:34: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.102 2 packetsJun 8 10:40:35: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.102 2 packetsJun 8 10:44:36: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.104.143 2 packetsJun 8 10:56:36: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.104.143 2 packetsJun 8 11:03:36: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.102 2 packetsJun 8 11:31:38: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packetsJun 8 12:02:39: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.102 2 packetsJun 8 13:24:42: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packetsJun 8 13:32:43: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 2 packetsJun 8 14:05:33: %HSRP-5-STATECHANGE: Vlan99 Grp 2 state Standby -> Active
Jun 8 14:05:33: %HSRP-5-STATECHANGE: Vlan99 Grp 2 state Active -> Speak
Jun 8 14:05:44: %HSRP-5-STATECHANGE: Vlan99 Grp 2 state Speak -> Standby
Jun 8 14:07:44: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 2 packetsJun 8 14:50:46: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 1 packetJun 8 14:55:46: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 1 packetJun 8 15:09:20: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.46 1 packetJun 8 15:14:47: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.46 11 packetsJun 8 15:19:47: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.46 14 packetsJun 8 15:29:48: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.46 30 packetsJun 8 15:36:48: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.46 2 packetsJun 8 15:43:49: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 2 packetsJun 8 15:47:49: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.93 2 packetsJun 8 15:48:49: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 2 packetsJun 8 16:10:50: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.46 2 packetsJun 8 16:18:35: %HSRP-5-STATECHANGE: Vlan2 Grp 2 state Standby -> Active
Jun 8 16:18:39: %HSRP-5-STATECHANGE: Vlan2 Grp 2 state Active -> Speak
Jun 8 16:18:50: %HSRP-5-STATECHANGE: Vlan2 Grp 2 state Speak -> Standby
Jun 8 16:26:50: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 4 packetsJun 8 17:02:52: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.104.143 2 packetsJun 8 17:50:54: %SEC-6-IPACCESSLOGS: list 99 permitted 10.232.106.46 4 packetsJun 8 18:29:56: %SEC-6-IPACCESSLOGS: list 99 denied 24.57.129.202 2 packets