Strange issue - 4451-X, layer 2 fiber WAN

[netdawg]

Hello,

I am trying to bring up a point to point fiber WAN between two datacenters. I have a 4451-X at each end, with GLC-LH-SMD SFP.

The physical link is up, however the routers can not ping each other. The telco providing the link assigned an IP to their interface, and was able to ping both our routers on each end.

Router1 - Switch - OK

Switch - Router1 - OK

Router2 - Switch - OK

Switch - Router2 - OK

Router1 - Router2 - Not ok

Router2 - Router1 - Not ok

The routers are new installs, so have a vanilla configuration, with only the IP address assigned to the proper gi0/0/2 SFP ports.

Now the fun part. Router2 can see the mac address of Router1 in it's arp table. Router1 shows incomplete.

I ran debug arp on both devices, while pinging, and I see arp request come from Router1 to Router2, and Router2 sends a response. But that response never makes it back to Router1.

Telco is suggesting we have either a faulty SFP, or dirty fiber. But how could that be possible if their switch can ping Router1 and Router2?

[Georg Pauwen]

Do you have someone at the Toronto office that can swap the cables and reseat the SFP ?

[netdawg]

It is in a datacenter, but I can have one of the DC techs do it I guess. My concern is that it is an LC patch cable, might not be so easy to swap. They should be easily able to reset the SFP as well.

[netdawg]

DC tech has tried re-seating the SFP, swapping the SFP, and changing polarity on the cable. When we changed polarity, we lost the comms with the telco switch, I was unable to ping it or my router. Changed polarity back, and I was back to being able to ping the switch, but not my remote router.

[Leo Laohoo]

---

@netdawg wrote:

I am running IOS-XE 16.09.01 .

---

Raise a TAC Case.  I am suspecting the issue is due to the IOS-XE version. 

There are several known SFP/SFP+ issues regarding 16.X.1 to 16.X.3.   Things like traffic gets dropped to ports staying down.  

[netdawg]

They (telco) finally resolved this.

They changed the path to use Toronto-Vancover-Seattle-Dallas instead of Toronto-Chicago-Dallas, and it worked. Exact same hardware on both their NNIs. Cisco in Canada, Juniper in USA, going to Alcatel-Lucent switching.

The Vancouver link worked, so they started checking configs on everything, and found one difference on the Alcatel-Lucent switches.

Something about a tag 0 versus tag * (wildcard). This was causing one way traffic essentially. They changed the Toronto-Chicago one to tag * instead of tag 0, and it worked.

I can now ping across, view each router via CDP, etc.

Thanks for all the feedback :)

[paul driver]

Hello

Kudos to you techs for finding that out, And thank for you for informing us of this resolution -  very much appreciated!

[Georg Pauwen]

Hello,

swap the fiber strands (transmit/receive) on the SFP...

Also, post the output of:

show inventory

[Leo Laohoo]

What firmware are the routers running on?