Last week We had a strange occurrence at work. We have 3 ISP's that are all doing BGP to advertise our prefixes. Tuesday Morning at 6:04 our biggest connection went down. Good thing was the others took over. This is where things went weird.
Contacted AT&T referencing the auto generated ticket from them. After 4 days they replace a bunch of bright jacks (Fancy ATT stuff to help them monitor light.). Various SFP's, XFP's and cross connects. Or so I'm told. The connection was still down.
When i was on my router which is running Cisco IOS XE Software, Version 03.16.03.S - Extended Support Release Cisco IOS Software, ASR1000 Software (PPC_LINUX_IOSD-ADVENTERPRISEK9-M), Version 15.5(3)S3, RELEASE SOFTWARE (fc1). I couldn't ping the interface on my other end. When I checked ARP for the IP's my side was listed correctly but the other end showed incomplete. They stated they could see mine. Finally we decided to reboot our router and when the router came back up it saw the mac address on the other end and brought the connection back up.
This to me doesn't add up. I reseated the XFP, reseated the 10GB SPA card that had the connection. We even tried other XFP's but the connection wouldn't go up. I don't have any logs. To me this sounds like a strange bug but I don't know what ATT did on the other end yet.
Has anyone seen this before? The connection that went down was running for 2 months prior to going down. The connection is also configured to be doing primary and secondary encapsulation (Q on Q) which I had never seen before.
