3G - Lose 50% of pings to LAN device but not to SVI

steven.crutchley · ‎09-09-2014

Hi have a 3G 887 Router (standalone) connected into an MPLS network.

Fa0 is connected to the local LAN on VLAN1 (192.168.1.254/24).
Dialer 1 is the default route. There are no other connections in the router.
There is a PC on the local LAN (192.168.1.10)

There is a remote MPLS site with LAN range 172.16.2.0/24.

If I ping from the remote MPLS site to the VLAN1 interface all pings respond ok.
If I ping from the remote MPLS site to PC on site 50% of the pings fail (alternating).
If I ping from the PC on site to the MPLS site 50% of the pings fail (alternating).

When I do packet debugging I find that only the pings that are getting through are showing up:

Ping from remote site to PC - 2/5 pings succeed (.!.!.)

Sep 9 17:40:13 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254, len 60, input feature
Sep 9 17:40:13 BST:     ICMP type=8, code=0, Common Flow Table(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:13 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254, len 60, input feature
Sep 9 17:40:13 BST:     ICMP type=8, code=0, Stateful Inspection(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:13 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254, len 60, input feature
Sep 9 17:40:13 BST:     ICMP type=8, code=0, MCI Check(101), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:13 BST: FIBipv4-packet-proc: route packet from Vlan1 src 192.168.1.10 dst 172.16.2.254
Sep 9 17:40:13 BST: FIBfwd-proc: packet routed by adj to Dialer1 0.0.0.0
Sep 9 17:40:13 BST: FIBipv4-packet-proc: packet routing succeeded
Sep 9 17:40:13 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Dialer1), len 60, output feature
Sep 9 17:40:13 BST:     ICMP type=8, code=0, Dialer idle reset(97), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:13 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Dialer1), len 60, output feature
Sep 9 17:40:13 BST:     ICMP type=8, code=0, Dialer idle reset(98), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:13 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Dialer1), g=172.16.2.254, len 60, forward
Sep 9 17:40:13 BST:     ICMP type=8, code=0
Sep 9 17:40:13 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Cellular0), len 60, sending full packet
Sep 9 17:40:13 BST:     ICMP type=8, code=0
Sep 9 17:40:19 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254, len 60, input feature
Sep 9 17:40:19 BST:     ICMP type=8, code=0, Common Flow Table(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:19 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254, len 60, input feature
Sep 9 17:40:19 BST:     ICMP type=8, code=0, Stateful Inspection(7), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:19 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254, len 60, input feature
Sep 9 17:40:19 BST:     ICMP type=8, code=0, MCI Check(101), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:19 BST: FIBipv4-packet-proc: route packet from Vlan1 src 192.168.1.10 dst 172.16.2.254
Sep 9 17:40:19 BST: FIBfwd-proc: packet routed by adj to Dialer1 0.0.0.0
Sep 9 17:40:19 BST: FIBipv4-packet-proc: packet routing succeeded
Sep 9 17:40:19 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Dialer1), len 60, output feature
Sep 9 17:40:19 BST:     ICMP type=8, code=0, Dialer idle reset(97), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:19 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Dialer1), len 60, output feature
Sep 9 17:40:19 BST:     ICMP type=8, code=0, Dialer idle reset(98), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Sep 9 17:40:19 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Dialer1), g=172.16.2.254, len 60, forward
Sep 9 17:40:19 BST:     ICMP type=8, code=0
Sep 9 17:40:19 BST: IP: s=192.168.1.10 (Vlan1), d=172.16.2.254 (Cellular0), len 60, sending full packet
Sep 9 17:40:19 BST:     ICMP type=8, code=0

If I run wireshark on the PC and ping it from the remote site, I see the incoming ICMP requests but only get half of them back.

This indicates that whatever is happening as traffic enters fa0 is causing it to be dropped before it even hits the ACL to show up.

Successful and failed pings always show up in an alternating format.

Can anyone advise what might be causing this?

steven.crutchley · ‎11-01-2014

I have fixed the problem. I believe to to be an IOS bug.

The solution appears to be very strange.

You need to create an ACL that permits anything AND applies logging (it will not work without logging) inbound on the vlan1 interface.

access-list 101 permit ip any any log

int vlan 1

ip access-group 101 in

Why this fixes the issue I am unsure. But leaving no ACL on the interface will still cause the router to suffer the 50% loss issue. To make it weirder logging on the ACL must be on.

tadeystas · ‎10-26-2016

I had a similar problem. I was losing every other packet. I tried access list to permit all, and it works. Have no idea how or why!

Is it because now it gets switched by CPU (process switching) instead of CEF, since we applied access-list? Please, comment with any ideas/suggestions.

travd · ‎06-16-2019

I also had this issue, thank you for posting, I have applied the open ACL and now everything is working, no more packet loss. This certainly is a weird one, I'm going to try a firmware update and see if it also fixes the problem.