ISR router cannot receive packets addressed to itself?

Maciex · ‎08-04-2014

Hello, Support Team and All Members,
I have a C881G router connected to 2 different ISP networks with a failover function configured and running properly. The following is a simple network diagram:

The main WAN traffic goes through the ISP 1 LTE network and the router, provided by that ISP. The DMS Host on that router points to our C881G router Fa4 WAN interface (192.168.1.10), so the ISP 1 NAT Router is practically transparent to our traffic. Our C881G tracks the DNS server within the ISP 1 network (194.dns.isp.1) and in case of it's inaccessibility the traffic is switched to the backup link, served by the on-board HSPA+ modem (interface Dialer0 of our C881G), connected to the ISP 2 HSPA network. It works fine, but the problem is with the PPTP connections from outside to the C881G router. The PPTP calls work always from the PPTP Client 2 PC (directly connected to the Fa4 subnet), but from PPTP Client 1 PC it works only in the failover mode - when all traffic goes through the ISP 2. The incoming path via ISP 1 does not work. The problem is rather not connected to the PPTP VPN, GRE, authentication or encryption, because just the first TCP 1723 SYN packets are dropped at Fa4 much earlier by the C881G router. The debug ip packet detail shows the following routing decision:

IP: s=194.xxx.yyy.80 (FastEthernet4), d=192.168.1.10, len 40, input feature
TCP src=4241, dst=1723, seq=791503628, ack=4111924253, win=0 ACK RST, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
FIBipv4-packet-proc: route packet from FastEthernet4 src 194.xxx.yyy.80 dst 192.168.1.10
FIBfwd-proc: Default:192.168.1.10/32 receive entry
FIBipv4-packet-proc: packet routing failed

All other packets addressed from outside networks to the router itself and received via the Fa4 are also dropped in this way. All packets sent to Fa4 from the local subnet 192.168.1.0 are accepted. The routing table shows only standard connected interfaces and 1 static route to the 194.dns.isp.1 via 192.168.1.1, which is also the tracked gateway of last resort.

Router runs the CEF.

I cannot locate in the following configuration file any statement preventing the packets addressed to the router itself:

___________________________________________________

!
version 15.3
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service internal
!
hostname C881_xyz
!
boot-start-marker
boot-end-marker
!
!
logging buffered 8192
no logging console
no logging monitor
!
no aaa new-model
clock timezone PCTime 1 0
clock summer-time PCTime date Mar 30 2003 2:00 Oct 26 2003 3:00
!
crypto ...
... <removed for sanity>
crypto pki ...
!
ip dhcp excluded-address 192.168.70.1 192.168.70.99
ip dhcp excluded-address 192.168.70.180 192.168.70.254
ip dhcp excluded-address 192.168.71.1 192.168.71.99
ip dhcp excluded-address 192.168.71.180 192.168.71.254
!
ip dhcp pool ccp-pool
import all
network 192.168.70.0 255.255.255.0
dns-server 8.8.8.8 8.8.4.4
default-router 192.168.70.1
lease 0 12
!
ip dhcp pool NVR
import all
network 192.168.71.0 255.255.255.0
dns-server 8.8.8.8 8.8.4.4
default-router 192.168.71.1
lease 0 12
!
!
ip domain name mydomain.com
ip name-server 8.8.8.8
ip name-server 8.8.4.4
ip inspect WAAS flush-timeout 10
ip cef
no ipv6 cef
!
!
multilink bundle-name authenticated
vpdn enable
!
vpdn-group 1
! Default PPTP VPDN group
accept-dialin
protocol pptp
virtual-template 1
!
chat-script gsm "" "AT!SCACT=1,1" TIMEOUT 60 "OK"
license udi pid C881G+7-K9 sn ***********
!
username admin privilege 15 secret 5 ******************************
!
controller Cellular 0
!
track 1 ip sla 1 reachability
delay down 1 up 30
!
interface FastEthernet0
description All VLANs Trunk
switchport mode trunk
no ip address
!
interface FastEthernet1
description VLAN 1 - LAN Main
no ip address
!
interface FastEthernet2
description VLAN 20 - LAN NVR
switchport access vlan 20
no ip address
!
interface FastEthernet3
description Traffic Monitoring only
no ip address
!
interface FastEthernet4
description WAN SP1$ETH-WAN$
ip address 192.168.1.10 255.255.255.0
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface Virtual-Template1
ip unnumbered FastEthernet4
peer default ip address pool vpn_pptp_pool
no keepalive
ppp encrypt mppe auto
ppp authentication ms-chap-v2
!
interface Cellular0
ip address negotiated
ip nat outside
ip virtual-reassembly in
encapsulation slip
dialer in-band
dialer pool-member 1
dialer-group 1
async mode interactive
!
interface Vlan1
description LAN Main
ip address 192.168.70.1 255.255.255.0
ip flow ingress
ip flow egress
ip nat inside
ip virtual-reassembly in
!
interface Vlan20
description LAN NVR
ip address 192.168.71.1 255.255.255.0
ip flow ingress
ip flow egress
ip nat inside
ip virtual-reassembly in
!
interface Dialer0
ip address negotiated
ip nat outside
ip virtual-reassembly in
encapsulation slip
dialer pool 1
dialer idle-timeout 0
dialer string gsm
dialer persistent
dialer-group 1
!
ip local policy route-map track-primary-if
ip local pool vpn_pptp_pool 192.168.70.180 192.168.70.199
ip forward-protocol nd
no ip http server
ip http access-class 1
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
ip flow-top-talkers
top 32
sort-by bytes
cache-timeout 600000
!
ip nat inside source route-map ISP_1 interface FastEthernet4 overload
ip nat inside source route-map ISP_2 interface Dialer0 overload
ip route 0.0.0.0 0.0.0.0 192.168.1.1 track 1
ip route 0.0.0.0 0.0.0.0 Dialer0 253
ip route 194.dns.isp.1 255.255.255.255 192.168.1.1
!
ip sla auto discovery
ip sla 1
icmp-echo 194.dns.isp.1 source-interface FastEthernet4
frequency 10
ip sla schedule 1 life forever start-time now
logging trap debugging
dialer-list 1 protocol ip permit
!
route-map track-primary-if permit 1
match ip address 100
set interface FastEthernet4
!
route-map Static_ISP_2 permit 10
match interface Dialer0
!
route-map Static_ISP_1 permit 10
match interface FastEthernet4
!
route-map ISP_2 permit 10
match ip address 1
match interface Dialer0
!
route-map ISP_1 permit 10
match ip address 1
match interface FastEthernet4
!
access-list 1 remark List for outside NATs
access-list 1 remark CCP_ACL Category=2
access-list 1 permit 192.168.70.0 0.0.0.255
access-list 1 permit 192.168.71.0 0.0.0.255
access-list 100 remark CCP_ACL Category=0
access-list 100 permit icmp any host 194.dns.isp.1
access-list 105 remark List for debugging local ICMP tests
access-list 105 remark CCP_ACL Category=16
access-list 105 permit icmp any any
!
control-plane
!
line con 0
no modem enable
line aux 0
line 3
script dialer gsm
modem InOut
no exec
transport input all
rxspeed 21600000
txspeed 5760000
line vty 0 4
exec-timeout 0 0
privilege level 15
login local
transport input telnet ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input telnet ssh
!
!
ntp update-calendar
ntp server 195.time.srv.1
!
end

___________________________________________________

Do you have an idea what can be the reason of that behaviour?

I really appreciate your suggestions,

Maciex

Peter Paluch · ‎08-04-2014

Hello Maciex,

I am afraid that the debug ip packet detailed has led you to a wrong conclusion. Whatever the "forus FALSE" means, it does not indicate that the router refuses to consider the packet as addressed to itself. I've just concocted a very quick test - two routers connected back to back, one is 10.0.1.1/24, the other is 10.0.1.2/24. I am pinging 10.0.1.2 from 10.0.1.1 and this is what 10.0.1.2 shows me:

*Aug  4 23:09:38.067: IP: s=10.0.1.1 (Ethernet2/1), d=10.0.1.2, len 100, input feature
*Aug  4 23:09:38.071:     ICMP type=8, code=0, MCI Check(94), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Aug  4 23:09:38.079: FIBipv4-packet-proc: route packet from Ethernet2/1 src 10.0.1.1 dst 10.0.1.2
*Aug  4 23:09:38.083: FIBfwd-proc: Default:10.0.1.2/32 receive entry
*Aug  4 23:09:38.083: FIBipv4-packet-proc: packet routing failed
*Aug  4 23:09:38.087: IP: tableid=0, s=10.0.1.1 (Ethernet2/1), d=10.0.1.2 (Ethernet2/1), routed via RIB
*Aug  4 23:09:38.091: IP: s=10.0.1.1 (Ethernet2/1), d=10.0.1.2 (Ethernet2/1), len 100, rcvd 3
*Aug  4 23:09:38.095:     ICMP type=8, code=0
*Aug  4 23:09:38.099: IP: s=10.0.1.1 (Ethernet2/1), d=10.0.1.2, len 100, stop process pak for forus packet
*Aug  4 23:09:38.103:     ICMP type=8, code=0
*Aug  4 23:09:38.107: FIBipv4-packet-proc: route packet from (local) src 10.0.1.2 dst 10.0.1.1
*Aug  4 23:09:38.111: FIBfwd-proc: packet routed by adj to Ethernet2/1 10.0.1.1
*Aug  4 23:09:38.111: FIBipv4-packet-proc: packet routing succeeded
*Aug  4 23:09:38.115: IP: s=10.0.1.2 (local), d=10.0.1.1 (Ethernet2/1), len 100, sending
*Aug  4 23:09:38.119:     ICMP type=0, code=0
*Aug  4 23:09:38.127: IP: s=10.0.1.2 (local), d=10.0.1.1 (Ethernet2/1), len 100, sending full packet
*Aug  4 23:09:38.131:     ICMP type=0, code=0

Note that even here, the router said the same as yours - and yet it did respond successfully to the ping request.

There is, I am afraid, a more mundane problem. PPTP is generally incompatible with PAT. PPTP uses two data streams: one is the control channel run over TCP port 1723, the other is the actual tunneled traffic - however, that traffic is essentially GRE-encapsulated, put directly into IP packets with no port information (there is no TCP/UDP involved). Without special support on the ISP 1 NAT box, PPTP sessions will not be able to pass through it. You will have to negotiate this with your ISP 1 - ask him to configure its NAT box with PPTP Application Layer Gateway support and allow IP protocol 47 (GRE).

This would explain why the PPTP Client 2 can always connect to your router - it is because there is no NAT/PAT/FW between the client and the router. It would also explain why Client 1 is able to connect over ISP 2 - because on that path, there is no NAT/PAT/FW box apparently present and there is a direct connectivity to the public IP address of your router.

Try talking to your ISP 1 about this.

Best regards,
Peter

Maciex · ‎08-05-2014

Hello, Peter,

Thank you for your attention and for the advice – definitely the “debug ip packet detailed” does not present the real router’s reaction here. Your experiment shows that the router can reply to the ICMP echo request regardless of the previous “packet routing failed” debug message, so that message seems worthless.

But your router finally accepted the incoming ICMP echo request and replied with the ICMP echo reply packet, as we can see in the further part of your log. My problem is that our router does not accept and does not reply to the TCP 1723 SYN packets at all. I use PPTP for several years in tens of different installations and I am familiar with the numerous problems with the devices unable to transport the GRE packets. But in our scenario the same physical ISP 1 Router (with exactly the same configuration) was used before we installed the new C881G router – previously we used a Cisco RV320 router in that place with only one ISP 1 service. The RV320 router was unstable and used to drop packets from time to time (there are many similar complains on our forum) but the PPTP VPN worked fluently with the RV320 all the time. For me it is a proof that the “DMZ Host” function on the ISP 1 Router works properly – not only for TCP and UDP but also for GRE protocol. I have an administrative access to that simple ISP 1 Router and there are no complex parameters for the DMZ Host function – just “Enable” checkbox and a target IP address form field.

The typical PPTP session begins with at least 9 TCP port 1723 packets exchanged in both directions, including TCP three-way handshake, the PPTP call request and reply and the link info, before any GRE packets start. In our case the C881G router does not reply to the incoming TCP 1723 packet with the SYN/ACK packet so the preceding TCP connection is never opened. We can see in the debug log that the incoming packet is delivered to the Fa4 interface but it is not further processed for some reason. The router also does not reply to TCP port 22 and TCP port 443 packets delivered to Fa4 from outside ISP 1, even does not reply with the RST/ACK packets. On the other hand the C881G router properly forwards packets from Fa4 to the VLAN subnet when we turn on the PAT for tests using the “ip nat inside source static tcp” command. Only the packets addressed to the router itself are not forwarded. Do you have an idea, what debug command could provide more information about the reason of that behavior?

Best Regards,

Maciex

Peter Paluch · ‎08-14-2014

Hi Maciex,

I sincerely apologize for responding lately.

If this issue is still unresolved, would you first please post the output of the following commands from the C881 router?

show control-plane host open-ports
show ip nat translations | i 1723

In addition, would you mind running the following debugs on the C881 router?

debug ip tcp transactions
debug vpdn l2x events
debug vpdn l2x errors

After these debugs are activated, try creating the PPTP tunnel over the ISP1 to the C881. Be sure to capture all debugs in details and please post them here. This might get us the necessary clues.

Best regards,
Peter