Recently we upgraded our data center switch performing the following tasks. The upgrade was driving by a faulty power supply slot preventing high input to the redundant power supplies. I will describe before and after.
The original config was a 6509, single Sup II MSFC, 2 10/100-48 port blades, 2 6148, 1 6148A and 2 6408 fiber blades.
The upgrade replaced and upgraded the chassis to a 6509E, added a second Sup II MSFC, replaced the two 6148 with 6148A. The IOS Version is
12.2(18)SXF17. The second Sup II is running in redundant mode and has no fiber connections on it.
Almost immediately after the upgrade, minor performance issues began to appear. VPN sessions from a 3020 Concentrator connected to FE ports began to drop RDP sessions to both VM servers on directly attached blade centers, as well as workstations on the LAN. Those workstations are on a 6509, connected via etherchannel fiber. Additionally, monitoring systems began randomly dropping packets to routers directly connected to Fast E ports on blades 3/4. The problem begins manifesting itself after 6pm in the evening when backup processes begin. Prior to the chassis upgrade, these issues were not present.
There are no reported errors, high CPU or other reported issues either in the logs, or with Solar Winds NPM. I am asking for any advanced commands to assist in isolating and eliminating these issues. Is it possible that some new features need to be enabled on the Sup to utilize the E chassis and backplane? Thanks in advance for any assistance in identifying this issue.
Do the problems only happen during the backups? Is the problem isolated to the same VLAN(s) as the servers kicking off the back up. If so, from the information provided so far I would suspect unicast flooding. A quick way to confirm is to check for output drops and uniform output rate on the switch.
- show interfaces | incl drops|rate|is up
The output drops would explain dropped packets impacting other services and the almost uniform rate for most ports would confirm flooding in the VLAN. To determine the src/dst of the traffic flooding stick a sniffer to an access port in the suspect VLAN(s). You will see unicast flow(s) on the port confirming the issue.
There is an excellent whitepaper on the subject that covers all the details.
The most common cause for unicast flooding is one of the following:
1. Assymetrical Routing
2. Spanning-Tree instability
3. CAM table exhaustion
By far the most common cause is Assymentrical Routing that may have been introduced due to a change in the path after the chassis was swapped out. The best way to resolve unicast flooding due to assymetrical routing is to raise your mac aging timers to match ARP (14,200 secs).
If STP instablility is the problem you will see a high amount of TCNs reported.
- show spanning-tree det | incl ieee|occurr|from|is exec
GoalDocumentationDefineAdd Device to Smart AccountSync Smart Account via vManage1.1 VNF package for vBranchDesignDeployOperate
To successfully provision a ENCS device in remote site with internet connection.
Minimum software relea...
はじめに確認方法Version による Application name の変更について備考参考情報 はじめに本ドキュメントでは Cisco SD-WAN における Policy 上で設定可能な Application を確認する方法について記載しています。 確認方法サポートされている Application name についてはご使用されている vManage へ API を呼び出して確認することが可能です。https://<IP or FQDN>/...
DMVPN (Dynamic Multipoint VPN) Introduced by Cisco in late 2000 is a routing technology you can use to build a VPN network with multiple sites (spokes) without having to statically configure all devices. It’s a “hub and spoke” network, where the spok...
On 24th August 2021, Cisco announced the latest IOS XE release - Cisco IOS XE Bengaluru 17.6.1a
IOS XE 17.6.1a unlocks various routing features and enhancements comprehensively covering different technology segments such as voice, security,...
DMVPN (Dynamic Multipoint VPN) Introduced by Cisco in late 2000 is a routing technology you can use to build a VPN network with multiple sites (spokes) without having to statically configure all devices. It’s a “hub and spoke” network, where th...