We upgraded from sup 720s running to SUPT2s running VSS with VPLS and the VCs wouldn't pass traffic so we had to roll back. I am wondering if our procedure for the upgrade might have caused the problem. We have tested VPLS failing over on the SUP2Ts in the lab and it works great, however in the production network we had trouble.
When we upgraded from HSRP to VSS, we had a requirement to keep the outages brief. To achieve that we:
1) shut down all ports on the active HSRP switch (Switch 1). This was because the VPLS configuration was on the standby switch (Switch 2).
2) powered down switch 1 and installed the VSS configured SUP2T with all ports shut down.
3) waited until all modules were okay.
4) using scripts shut down all ports on switch 2, and immediately did a no shut on switch 1.
5) powered down switch 2 and installed the VSS configured SUP2T with all ports shut down
6) did a no shut on all ports on switch 2
This allowed us to have a minimal outage in the network. IP routing came back just fine.
After we upgraded all VPLS VCs showed that they were UP, but they showed one-way traffic and remote macs weren't learned.
I was wondering if this upgrade method somehow confused MPLS labeling or something in VPLS to cause it not to work? Maybe some tables related to related to MPLS or VPLS weren't flushed out due to the quick switch over?
After working with TAC I found out that for VPLS to recover from changes, you need to have
mpls ldp graceful-restart
configured on all devices using VPLS. If the command is entered after VPLS connections are already running, you will need to enter clear
mpls ldp neighbor x.x.x.x or *.
That will restart the ldp session which in turn will enable graceful restart for that session. Graceful restart handles changes for VPLS.
I'm planning to have a similar setup “VSS with MPLS/VPLS using Sup2T and WS-X69xx/WS-X68xx line cards” for green field deployment, however; I don’t have a time to PoC the proposed solution, I’d appreciate if you share any limitations/restrictions of deploying such scenario.
Also could you please advise if this is a validated solution by Cisco
Some line cards are supported and some aren't. See chart five for chassis, card and IOS compatibility.
This is the bug that provides the work around Validated by Cisco
CSCsw70062 Bug Details
checkpoint VC info to standby even in the absence of graceful restart
Xconnect traffic entering the Standby Switch of a VSS-pair with SUP-2Ts or through the Inactive Supervisor will not be forwarded to the CE-facing interfaces:
Workaround:Enable MPLS LDP Graceful Restart on all Routers / Switches with LDP Peers to this Switch / Router and clear the existing LDP sessions between the devices
I was out last week so it took me until today to respond. I hope it is soon enough to be helpful.
I updated my question with the answers you requested. Hopefully you were notified.
Professional Network Administrator
AT&T Global Services
Confidential: This e-mail and any files transmitted with it are the property of AT&T and/or its affiliates, are confidential, and are intended solely for the use of the individual or entity to which this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender at 760-715-4149 and delete this message immediately from your computer. Any other use, retention, dissemination forwarding, printing, or copying of this e-mail is strictly prohibited.
I experienced the same behavior with pseudowires on my 6504E/Sup2T's. I have LDP and CE interfaces present on both active and standby supervisors. EoMPLS traffic arriving on the standby supervisor would not be forwarded out the CE-facing interface.
Problem is that LDP fails to populate the standby supervisor's hardware forwarding tables like it does for the active sup and all the line cards. Putting MPLS LDP GRACEFUL-RESTART on the PE's and each of their neighbors fixed it as mentioned previously. Apparently, the GRACEFUL-RESTART code happens to touch the standby sup's forwarding tables.