Re: Strange issue - 4451-X, layer 2 fiber WAN

netdawg · ‎10-26-2018

Hello,

I am trying to bring up a point to point fiber WAN between two datacenters. I have a 4451-X at each end, with GLC-LH-SMD SFP.

The physical link is up, however the routers can not ping each other. The telco providing the link assigned an IP to their interface, and was able to ping both our routers on each end.

Router1 - Switch - OK

Switch - Router1 - OK

Router2 - Switch - OK

Switch - Router2 - OK

Router1 - Router2 - Not ok

Router2 - Router1 - Not ok

The routers are new installs, so have a vanilla configuration, with only the IP address assigned to the proper gi0/0/2 SFP ports.

Now the fun part. Router2 can see the mac address of Router1 in it's arp table. Router1 shows incomplete.

I ran debug arp on both devices, while pinging, and I see arp request come from Router1 to Router2, and Router2 sends a response. But that response never makes it back to Router1.

Telco is suggesting we have either a faulty SFP, or dirty fiber. But how could that be possible if their switch can ping Router1 and Router2?

balaji.bandi · ‎10-26-2018

what kind of switches and routers, do you have any network topology and post the configuration will be beneficial to suggest.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

netdawg · ‎10-29-2018

My device at each end is a 4451-X router, and the telco is using Ciena 5142 switch.

I am running IOS-XE 16.09.01 .

The topology is simple, Toronto 4451-X connects via GLC-LH-SMD and SM patch cable to the telco switch, where it then goes into "their cloud" until it comes out via a SM patch in Dallas to my other 4451-X.

Georg Pauwen · ‎10-29-2018

Are these genuine Cisco or third party SFPs ?

netdawg · ‎10-29-2018

As far as I know, both ends are genuine.

Which is why it doesn't make sense to me, that the switch in the middle can ping both my routers, and I can ping the switch from both ends, but the routers can't ping through the switch to each other.

Also to add, I have done a debug arp on both endpoints, and I see the Toronto end sending arp requests, which DO make it to Dallas. Dallas replies, but the reply never makes it back to the Toronto end.

Dallas:

*Oct 29 13:25:57.628: IP ARP: rcvd req src 10.10.10.2 0042.681c.1ee2, dst 10.10.10.1 GigabitEthernet0/0/3 tableid 0
*Oct 29 13:25:57.628: IP ARP: sent rep src 10.10.10.1 0042.681c.1e53,
                 dst 10.10.10.2 0042.681c.1ee2 GigabitEthernet0/0/3
*Oct 29 13:25:59.630: IP ARP: rcvd req src 10.10.10.2 0042.681c.1ee2, dst 10.10.10.1 GigabitEthernet0/0/3 tableid 0
*Oct 29 13:25:59.630: IP ARP: sent rep src 10.10.10.1 0042.681c.1e53,
                 dst 10.10.10.2 0042.681c.1ee2 GigabitEthernet0/0/3
*Oct 29 13:26:01.630: IP ARP: rcvd req src 10.10.10.2 0042.681c.1ee2, dst 10.10.10.1 GigabitEthernet0/0/3 tableid 0
*Oct 29 13:26:01.631: IP ARP: sent rep src 10.10.10.1 0042.681c.1e53,
                 dst 10.10.10.2 0042.681c.1ee2 GigabitEthernet0/0/3

Toronto:

tor-core-rtr-01#ping 10.10.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.1, timeout is 2 seconds:

*Oct 29 13:11:03.747: IP ARP: creating incomplete entry for IP address: 10.10.10.1 interface GigabitEthernet0/0/2 tableid 0
*Oct 29 13:11:03.748: IP ARP: sent req src 10.10.10.2 0042.681c.1ee2,
                 dst 10.10.10.1 0000.0000.0000 GigabitEthernet0/0/2.
*Oct 29 13:11:05.749: IP ARP: sent req src 10.10.10.2 0042.681c.1ee2,
                 dst 10.10.10.1 0000.0000.0000 GigabitEthernet0/0/2.
*Oct 29 13:11:07.750: IP ARP: sent req src 10.10.10.2 0042.681c.1ee2,
                 dst 10.10.10.1 0000.0000.0000 GigabitEthernet0/0/2.
*Oct 29 13:11:09.751: IP ARP: sent req src 10.10.10.2 0042.681c.1ee2,
                 dst 10.10.10.1 0000.0000.0000 GigabitEthernet0/0/2.
*Oct 29 13:11:11.752: IP ARP: sent req src 10.10.10.2 0042.681c.1ee2,
                 dst 10.10.10.1 0000.0000.0000 GigabitEthernet0/0/2.
Success rate is 0 percent (0/5)

Georg Pauwen · ‎10-29-2018

Hello,

post the config of (one of) the routers...

netdawg · ‎10-29-2018

Current configuration : 1984 bytes
!
! Last configuration change at 19:32:15 UTC Fri Oct 26 2018
!
version 16.9
service timestamps debug datetime msec
service timestamps log datetime msec
platform qfp utilization monitor load 80
no platform punt-keepalive disable-kernel-core
!
hostname xxx-C4451RT-01
!
boot-start-marker
boot system bootflash:isr4400-universalk9.16.09.01.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
enable secret 5 xxxxxxxxxxxx
enable password xxxxxx
!
no aaa new-model
no process cpu extended history
no process cpu autoprofile hog
!
ip admission watch-list expiry-time 0
!
!
!
!
!
!
!
!
!
!
subscriber templating
multilink bundle-name authenticated
!
!
!
!
!
license udi pid ISR4451-X/K9 sn xxxxxxxxxxxxxxxxxx
no license smart enable
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
!
!
!
redundancy
mode none
!
!
!
no cdp run
!
!
!
!
!
interface GigabitEthernet0/0/0
no ip address
shutdown
media-type rj45
negotiation auto
!
interface GigabitEthernet0/0/1
no ip address
shutdown
media-type rj45
negotiation auto
!
interface GigabitEthernet0/0/2
description "xxxxx RJ45 to xxxxxxxx"
ip address 172.16.10.9 255.255.255.252
media-type rj45
negotiation auto
!
interface GigabitEthernet0/0/3
description "xxxxx fiber to xxToronto"
ip address 10.10.10.1 255.255.255.248
media-type sfp
negotiation auto
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address 172.31.71.150 255.255.255.0
negotiation auto
no cdp enable
!
ip forward-protocol nd
no ip http server
no ip http secure-server
ip tftp source-interface GigabitEthernet0
ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 172.31.71.254
!
!
no service-routing capabilities-manager
!
!
snmp-server community public RO
!
!
control-plane
!
!
line con 0
exec-timeout 0 0
transport input none
stopbits 1
line aux 0
stopbits 1
line vty 0 4
password xxxxxxxxxx
login
!
!
!
!
!
!
end

Georg Pauwen · ‎10-29-2018

Do you have access to both routers ? If so, post the output of 'show ip route' from both routers...

netdawg · ‎10-29-2018

DAL-C4451RT-01#sh ip rou
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.10.10.0/29 is directly connected, GigabitEthernet0/0/3
L        10.10.10.1/32 is directly connected, GigabitEthernet0/0/3
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.10.8/30 is directly connected, GigabitEthernet0/0/2
L        172.16.10.9/32 is directly connected, GigabitEthernet0/0/2

I wiped the Toronto router,= to be sure it wasn't something in the config, and only configured the gi0/0/2 interface after doing a wr erase.

Router#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.10.10.0/29 is directly connected, GigabitEthernet0/0/2
L        10.10.10.2/32 is directly connected, GigabitEthernet0/0/2

Georg Pauwen · ‎10-29-2018

Hello,

it should obviously not be necessary, but for the sake of testing try and add static host routes:

ip route 10.10.10.2 255.255.255.255 GigabitEthernet0/0/3

ip route 10.10.10.1 255.255.255.255 GigabitEthernet0/0/2

netdawg · ‎10-29-2018

No go, I also tried adding static arp entries for each router on the other one, and that didn't help either. I also tried changing the IP addresses used, and still the same lack of communication.

The fiber ends are obviously correct since I an reach the telco switch on both ends, and he can reach my devices, so I am strongly believing it is something on the telco end causing this.

Georg Pauwen · ‎10-29-2018

Did you try to swap the transmit/receive ends ? Also maybe reseating the SFPs might be worth a try...

netdawg · ‎10-29-2018

This has been done as well. The telco completed their work and verified we have the correct connections.

paul driver · ‎10-29-2018

Hello

Whats the led status of the port? - blinking - solid - color ?
Do you see anything in CDP the other side of that link?

What does the interface statistics show?

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

netdawg · ‎10-29-2018

I just enabled CDP, and my Toronto router shows in Dallas cdp neighbour table, but Dallas does not show in the Toronto cdp neighbour table. It is almost like packets can get out of Toronto, but not back in.