Howdy
We added Site02 for redundancy and DR. Also added new SAN and ESX hosts.
The CPU on device Site01Server01 is not very happy when files are being transferred between the sites.
Since the setup is the same at both sites, why is the problem only at site01?
Drawings attached, any help appreciated.
Site02 is newer and we have two 3750X-24 with IOS 12.2(55)SE3 on ipservices, for site02server01; no problem.
Site01server01 is a stack of two 3750G-24PS with IOS 12.2(58)SE on ipservices.
Some output from site01server01 switch:
site01server01#sh proc cpu | e 0.00+.+0.00
CPU utilization for five seconds: 97%/17%; one minute: 90%; five minutes: 62%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 6224333 554508 11224 0.00% 0.25% 0.19% 0 Check heaps
12 203092 453904 447 0.15% 0.01% 0.00% 0 ARP Input
73 1515032 55326 27383 0.63% 0.07% 0.01% 0 Per-minute Jobs
74 97220 3278016 29 0.00% 0.05% 0.01% 0 Per-Second Jobs
76 629199 74578814 8 0.31% 0.22% 0.13% 0 RedEarth Tx Mana
77 151057 128968304 1 0.00% 0.08% 0.04% 0 RedEarth Rx Mana
82 2268113 172974758 13 1.43% 0.98% 0.68% 0 Fifo Error Detec
86 436894 3360808 129 0.15% 0.03% 0.00% 0 hrpc <- response
96 332508 96520608 3 0.00% 0.15% 0.08% 0 HLFM address lea
98 162516 96547766 1 0.00% 0.06% 0.03% 0 HLFM address ret
102 6498138 326733 19888 0.00% 0.23% 0.18% 0 HULC Tcam Memory
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
117 691357 40954329 16 0.31% 0.37% 0.22% 0 hpm main process
121 1996393 5241576 380 0.31% 0.40% 0.27% 0 hpm counter proc
122 948094 3853974 246 0.00% 0.04% 0.01% 0 HRPC pm-counters
128 43142 2563878 16 0.15% 0.01% 0.00% 0 hl3mm
147 270764 16236135 16 0.00% 0.10% 0.06% 0 Hulc Storm Contr
161 9288189 97322877 95 4.14% 4.67% 3.13% 0 Hulc LED Process
171 2199514 653502 3365 0.15% 0.15% 0.12% 0 HQM Stack Proces
172 3075305 2610329 1178 0.00% 0.07% 0.08% 0 HRPC qos request
184 80860 3253008 24 0.00% 0.02% 0.01% 0 PI MATM Aging Pr
217 244402 1546665 158 0.00% 0.05% 0.01% 0 CDP Protocol
223 156320 96520712 1 0.15% 0.07% 0.01% 0 IP ARP Retry Age
224 104212895 19557297 5328 65.07% 57.10% 38.66% 0 IP Input
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
238 13304837 98349065 135 0.47% 0.54% 0.39% 0 Spanning Tree
254 122562 5036044 24 0.15% 0.04% 0.00% 0 CEF: IPv4 proces
274 158370 7225103 21 0.00% 0.05% 0.02% 0 Marvell wk-a Pow
289 104174 6501504 16 0.00% 0.06% 0.01% 0 DHCPD Receive
294 83646 32680692 2 0.00% 0.06% 0.01% 0 MDFS RP process
302 942 237 3974 0.15% 0.86% 0.24% 2 SSH Process
305 229805 102322961 2 0.15% 0.10% 0.05% 0 MMON MENG
311 2318417 126503 18326 0.00% 0.41% 0.43% 0 crypto sw pk pro
317 65247 18691657 3 0.15% 0.02% 0.00% 0 PM Callback
318 412392 1063829 387 0.15% 0.01% 0.00% 0 VLAN Manager
357 111523 11221896 9 0.00% 0.08% 0.02% 0 IGMP Input
360 238682 24226160 9 0.63% 0.17% 0.08% 0 PIM Process
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
362 1499632 5631482 266 0.48% 0.16% 0.12% 0 IP SNMP
363 406557 2811398 144 0.32% 0.04% 0.00% 0 PDU DISPATCHER
364 2056115 2886865 712 0.32% 0.18% 0.15% 0 SNMP ENGINE
site01server01#show interfaces switching
Vlan1
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 5961479 597177546 3718046 432556707
Cache misses 0 - - -
Fast 6 570 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 433495 26009700 1984 119040
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan157 iLo Mgmt
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 68632 6044310 218556 18879116
Cache misses 0 - - -
Fast 0 0 3 316
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 11900 714000 27623 1657380
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 42 2520 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan165 Storage Subnet
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 8195 1171838 325747464 152512709543
Cache misses 0 - - -
Fast 0 0 5 584
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 8701 522060 10248 614880
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 11 660 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Vlan2002 InterDataCenter_PtP_Link
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 325906188 152530627679 302429 23589582
Cache misses 0 - - -
Fast 3 291 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 80 4800 81 4860
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
site01server01#show ip traffic
IP statistics:
Rcvd: 333774251 total, 4774621 local destination
0 format errors, 0 checksum errors, 8527 bad hop count
0 unknown protocol, 1569160 not a gateway
0 security failures, 0 bad options, 639471 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 639471 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
661 fragmented, 0 couldn't fragment
Bcast: 8383 received, 1 sent
Mcast: 888759 received, 510860 sent
Sent: 4363457 generated, 2366166154 forwarded
Drop: 163 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero
ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 2037 unreachable
988657 echo, 159 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 26487 unreachable, 170 echo, 988657 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 6 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 23080 total, 0 checksum errors, 314 no port
Sent: 23562 total
UDP statistics:
Rcvd: 3407912 total, 0 checksum errors, 1662 no port
Sent: 2813384 total, 33404 forwarded broadcasts
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh
EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total
PIMv2 statistics: Sent/Received
Total: 111042/75571, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 111042/75545
Join/Prunes: 0/26, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0
IGMP statistics: Sent/Received
Total: 54620/36609, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 36034/18584, Host Reports: 18586/18024, Host Leaves: 0/1
DVMRP: 0/0, PIM: 0/0
Queue drops: 0
OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 240568 total, 0 checksum errors
227568 hello, 14 database desc, 0 link state req
8731 link state updates, 4255 link state acks
Sent: 345228 total
341143 hello, 12 database desc, 6 link state req
542 link state updates, 3525 link state acks
ARP statistics:
Rcvd: 452702 requests, 1806 replies, 53 reverse, 0 other
Sent: 28170 requests, 11845 replies (1 proxy), 0 reverse
Drop due to input queue full: 0
site01server01#sh ip int vlan2002
Vlan2002 is up, line protocol is up
Internet address is 192.168.199.66/30
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is enabled
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP Null turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Check hwidb
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
site01server01#sh ip int vlan165
Vlan165 is up, line protocol is up
Internet address is 192.168.165.1/24
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper addresses are 192.168.1.10
192.168.164.79
192.168.1.79
192.168.235.79
Directed broadcast forwarding is enabled
Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13
224.0.0.5 224.0.0.6
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP Null turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Check hwidb
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
site01server01#sh ip int vlan157
Vlan157 is up, line protocol is up
Internet address is 192.168.157.1/24
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper addresses are 192.168.1.10
192.168.164.79
192.168.1.79
192.168.235.79
Directed broadcast forwarding is enabled
Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13
224.0.0.5 224.0.0.6
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP Null turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Check hwidb
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled site01server01#sh ip int vlan2002
Vlan2002 is up, line protocol is up
Internet address is 192.168.199.66/30
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is enabled
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP Null turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Check hwidb
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
site01server01#sh ip int vlan165
Vlan165 is up, line protocol is up
Internet address is 192.168.165.1/24
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper addresses are 192.168.1.10
192.168.164.79
192.168.1.79
192.168.235.79
Directed broadcast forwarding is enabled
Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13
224.0.0.5 224.0.0.6
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP Null turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Check hwidb
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
site01server01#sh ip int vlan157
Vlan157 is up, line protocol is up
Internet address is 192.168.157.1/24
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper addresses are 192.168.1.10
192.168.164.79
192.168.1.79
192.168.235.79
Directed broadcast forwarding is enabled
Multicast reserved groups joined: 224.0.0.1 224.0.0.2 224.0.0.22 224.0.0.13
224.0.0.5 224.0.0.6
Outgoing access list is not set
Inbound access list is not set
Proxy ARP is enabled
Local Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
IP Flow switching is disabled
IP CEF switching is enabled
IP CEF switching turbo vector
IP Null turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF
Router Discovery is disabled
IP output packet accounting is disabled
IP access violation accounting is disabled
TCP/IP header compression is disabled
RTP/IP header compression is disabled
Probe proxy name replies are disabled
Policy routing is disabled
Network address translation is disabled
BGP Policy Mapping is disabled
Input features: MCI Check
Output features: Check hwidb
WCCP Redirect outbound is disabled
WCCP Redirect inbound is disabled
WCCP Redirect exclude is disabled
Figure 0.1 Physical
Figure 0.2 Logical
