Re: Strange routing

Alexandr Matskevich · ‎05-07-2013

Hello dear community!

I have 3825 with IOS version c3825-adventerprisek9_ivs-mz.151-4.M6.bin.

My ISP give me tagged Ethernet link with 3 Vlan ID and 3 static ip-addresses. It was created 3 pppoe-client on my router.

When I ping from Internet one of my static IP-address I don't get a response. I see these in terminal:

May 7 18:47:02.993: IP: s=A.A.83.246 (local), d=B.B.117.177, len 60, local feature

May 7 18:47:02.993: ICMP type=0, code=0, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

May 7 18:47:02.993: FIBipv4-packet-proc: route packet from (local) src A.A.83.246 dst B.B.117.177

May 7 18:47:02.993: FIBfwd-proc: Default:0.0.0.0/0 process level forwarding

May 7 18:47:02.993: FIBfwd-proc: depth 0 first_idx 0 paths 1 long 0(0)

May 7 18:47:02.997: FIBfwd-proc: try path 0 (of 1) v4-rcrsv-X.X.160.239 first short ext 0(-1)

May 7 18:47:02.997: FIBfwd-proc: v4-rcrsv-X.X.160.239 valid

May 7 18:47:02.997: FIBfwd-proc: ip_pak_table 0 ip_nh_table 0 if none nh X.X.160.239 deag 0 chg_if 0 via fib 69A199EC path type recursive

May 7 18:47:02.997: FIBfwd-proc: depth 1 first_idx 0 paths 3 long 0(0)

May 7 18:47:02.997: FIBfwd-proc: try path 0 (of 3) v4-ah-X.X.

C3825#unde all160.239-Di14 first short ext 0(-1)

May 7 18:47:02.997: FIBfwd-proc: v4-ah-X.X.160.239-Di14 valid

May 7 18:47:02.997: FIBfwd-proc: ip_pak_table 0 ip_nh_table 65535 if Dialer14 nh X.X.160.239 deag 0 chg_if 0 via fib 0 path type attached host

May 7 18:47:02.997: FIBfwd-proc: packet routed to Dialer14 X.X.160.239(0)

May 7 18:47:02.997: FIBipv4-packet-proc: packet routing succeeded

May 7 18:47:02.997: FIBfwd-proc: ip_pak_table 0 ip_nh_table 65535 if Dialer14 nh X.X.160.239 uhp 1 deag 0 ttlexp 0

May 7 18:47:02.997: FIBfwd-proc: sending link IP ip_pak_table 0 ip_nh_table 65535 if Dialer14 nh X.X.160.239 uhp 1 deag 0 chgif 0 ttlexp 0 rec 0

May 7 18:47:02.997: IP: s=A.A.83.246 (local), d=B.B.117.177 (Dialer14), len 60, sending

May 7 18:47:02.997: ICMP type=0, code=0

May 7 18:47:02.997: IP: s=A.A.83.246 (local), d=B.B.117.177 (Dialer14), len 60, output feature

May 7 18:47:02.997: ICMP type=0, code=0, Post-routing NAT Outside(24), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

Ping comes into Dialer12 and response routed to Dialer14. Why? How I can solve it?

My routing table:

C3825#sh ip route

Gateway of last resort is X.X.160.239 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via X.X.160.239

A.0.0.0/32 is subnetted, 3 subnets

C A.A.81.249 is directly connected, Dialer16

C A.A.83.202 is directly connected, Dialer14

C A.A.83.246 is directly connected, Dialer12

192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.10.0/24 is directly connected, Vlan1

L 192.168.10.162/32 is directly connected, Vlan1

192.168.17.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.17.0/24 is directly connected, Vlan17

L 192.168.17.1/32 is directly connected, Vlan17

X.X.160.0/32 is subnetted, 1 subnets

C X.X.160.239 is directly connected, Dialer16

is directly connected, Dialer14

is directly connected, Dialer12

C3825#sh ip ro X.X.160.239

Routing entry for X.X.160.239/32

Known via "connected", distance 0, metric 0 (connected, via interface)

Routing Descriptor Blocks:

directly connected, via Dialer16

Route metric is 0, traffic share count is 1

directly connected, via Dialer14

Route metric is 0, traffic share count is 1

* directly connected, via Dialer12

Route metric is 0, traffic share count is 1

And my ip interfaces:

C3825(config)#do s

Interface IP-Address OK? Method Status Protocol

GigabitEthernet0/0 unassigned YES NVRAM up up

GigabitEthernet0/0.12 unassigned YES unset up up

GigabitEthernet0/0.14 unassigned YES unset up up

GigabitEthernet0/0.16 unassigned YES unset up up

GigabitEthernet0/1 unassigned YES NVRAM administratively down down

FastEthernet0/0/0 unassigned YES unset up up

Dialer12 A.A.83.246 YES IPCP up up

Dialer14 A.A.83.202 YES IPCP up up

Dialer16 A.A.81.249 YES IPCP up up

NVI0 unassigned YES unset administratively down down

Virtual-Access1 unassigned YES unset down down

Virtual-Access2 A.A.83.246 YES unset up up

Virtual-Access3 unassigned YES unset up up

Virtual-Access4 unassigned YES unset up up

Virtual-Access5 unassigned YES unset up up

Virtual-Access6 unassigned YES unset up up

Virtual-Template1 A.A.83.246 YES unset down down

Vlan1 192.168.10.162 YES NVRAM up up

Vlan17 192.168.17.1 YES NVRAM up up

antonio.guirado · ‎05-07-2013

Hello,

This is is not strange. It only balancing.

Does ping work from your router?.

Are you using NAT in the Dialer interfaces?.

Does your ISP and your router support Multilink PPP?. Try to bundle all links in a only logical PPP link.

Regards

Alexandr Matskevich · ‎05-08-2013

How to disable this load balancing?

C3825#ping 8.8.8.8 so di 12

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with a source address of A.A.83.246

.....

Success rate is 0 percent (0/5)

C3825#ping 8.8.8.8 so di 14

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with a source address of A.A.83.202

.....

Success rate is 0 percent (0/5)

C3825#ping 8.8.8.8 so di 16

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with a source address of A.A.81.249

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 28/28/28 ms

I'm using NAT with PBR to route inside networks through Dialer 14 & Dialer16 interfaces. Mail server must be available throgh Dialer 16. Anyconnect must be available through Dialer12

My ISP not support Multilink PPP.

I suppose link bundling is not proper solution in my case. Correct me if I'm wrong.

antonio.guirado · ‎05-08-2013

I think you must contact your ISP. Ping must work. Check one by one your connection (doing a shutdow in the dialer interface) to check that each one works. I think in not a balancing problem. It is a connectivy problem.

Regards.

Alexandr Matskevich · ‎05-08-2013

C3825(config)#int di 14

C3825(config-if)#sh

C3825(config-if)#

May 8 11:44:06.544: %DIALER-6-UNBIND: Interface Vi5 unbound from profile Di14

May 8 11:44:06.544: Di14 DDR: dialer shutdown complete

May 8 11:44:06.548: %LINK-3-UPDOWN: Interface Virtual-Access5, changed state to down

C3825(config-if)#

May 8 11:44:06.552: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access5, changed state to down

C3825(config-if)#

May 8 11:44:08.544: %LINK-5-CHANGED: Interface Dialer14, changed state to administratively down

C3825(config-if)#do ping 8.8.8.8 so di

% Ambiguous command: "do ping 8.8.8.8 so di"

C3825(config-if)#do ping 8.8.8.8 so dialer12

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with a source address of A.A.83.246

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 28/30/32 ms

C3825(config-if)#do ping 8.8.8.8 so dialer16

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with a source address of A.A.81.249

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 28/29/32 ms

C3825(config-if)#do sh ip route

May 8 11:44:28.656: %LINK-3-UPDOWN: Interface Virtual-Access5, changed state to up

C3825(config-if)#do sh ip route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

+ - replicated route, % - next hop override

Gateway of last resort is B.B.160.239 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via B.B.160.239

A.0.0.0/32 is subnetted, 2 subnets

C A.A.81.249 is directly connected, Dialer16

C A.A.83.246 is directly connected, Dialer12

192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.10.0/24 is directly connected, Vlan1

L 192.168.10.162/32 is directly connected, Vlan1

192.168.17.0/24 is variably subnetted, 2 subnets, 2 masks

C 192.168.17.0/24 is directly connected, Vlan17

L 192.168.17.1/32 is directly connected, Vlan17

B.B.160.0/32 is subnetted, 1 subnets

C B.B.160.239 is directly connected, Dialer16

is directly connected, Dialer12

C3825(config-if)#do sh ip route B.B.160.239

Routing entry for B.B.160.239/32

Known via "connected", distance 0, metric 0 (connected, via interface)

Routing Descriptor Blocks:

* directly connected, via Dialer16

Route metric is 0, traffic share count is 1

directly connected, via Dialer12

Route metric is 0, traffic share count is 1

C3825(config-if)#

When I shutdown Dialer14 I can ping 8.8.8.8 through Dialer12 and through Dialer16. When I do "no shutdown" in interface Dialer14 configuration mode there's no ICMP reply from 8.8.8.8 to Dialer 12 and to Dialer 14, but Dialer 16 receive ICMP reply.

How to disable load balancing through multiple Dialers?

antonio.guirado · ‎05-08-2013

Hello,

I still think that is a routing problem not balancing. If you are using PBR you select (in the PBR) the next-hop

and if you are using NAT you select the return path because the IP address is only seen through one path. So

there is something else.

Try:

ping to x.160.239 from each dialer

shutdown dial12 and 16 and try icmp to 8.8.8.8 from Dialer14

one question. The connections are independent?. Each ip address x.83.246, x.83.202 and x.81.249 are supposed

to be associated to a different pppoe session?.

Regards

Alexandr Matskevich · ‎05-08-2013

Yes, connections are independent and each ip address is assigned on specified PPPoE session. I have one default route x.x.160.239 accessible through 3 Dialer interfaces with the same metric.

Current configuration : 10628 bytes

!

dot11 syslog

ip source-route

!

ip cef

!

ip name-server B.B.160.50

ip name-server B.B.160.65

no ipv6 cef

!

multilink bundle-name authenticated

!

redundancy

!

no ip ftp passive

ip ssh version 1

!

crypto isakmp policy 1

encr 3des

authentication pre-share

group 2

crypto isakmp key *** address *.*.*.*

!

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

!

crypto map ZYWALL_VPN 1 ipsec-isakmp

set peer *.*.*.*

set transform-set ESP-3DES-SHA

match address 105

!

interface GigabitEthernet0/0

description FIBER_WAN

no ip address

duplex auto

speed auto

media-type rj45

no cdp enable

!

interface GigabitEthernet0/0.12

encapsulation dot1Q 12

pppoe enable group global

pppoe-client dial-pool-number 12

!

interface GigabitEthernet0/0.14

encapsulation dot1Q 14

pppoe enable group global

pppoe-client dial-pool-number 14

!

interface GigabitEthernet0/0.16

encapsulation dot1Q 16

pppoe enable group global

pppoe-client dial-pool-number 16

!

interface FastEthernet0/0/0

description LAN VLP

switchport mode trunk

no ip address

spanning-tree portfast

!

interface Virtual-Template1

ip unnumbered Dialer12

!

interface Vlan1

ip address 192.168.10.162 255.255.255.0

ip nat inside

ip virtual-reassembly in

ip policy route-map USER_INTERNET

!

interface Vlan17

ip address 192.168.17.1 255.255.255.0

ip nat inside

ip virtual-reassembly in

ip policy route-map VOICE_INTERNET

!

interface Dialer12

mtu 1492

ip address negotiated

encapsulation ppp

dialer pool 12

dialer-group 12

ppp authentication chap callin

ppp chap hostname 111

ppp chap password 7 xxx

ppp ipcp dns accept

no cdp enable

crypto map ZYWALL_VPN

!

interface Dialer14

mtu 1492

ip address negotiated

ip nat outside

ip virtual-reassembly in

encapsulation ppp

shutdown

dialer pool 14

dialer-group 14

ppp authentication chap callin

ppp chap hostname 222

ppp chap password 7 xxx

ppp ipcp dns accept

!

interface Dialer16

mtu 1492

ip address negotiated

ip nat outside

ip virtual-reassembly in

encapsulation ppp

dialer pool 16

dialer-group 16

ppp authentication chap callin

ppp chap hostname 333

ppp chap password 7 xxx

ppp ipcp dns accept

!

ip local pool ANYCONNECT_POOL 192.168.11.1 192.168.11.254

ip forward-protocol nd

ip http server

ip http access-class 1

ip http authentication local

ip http secure-server

ip http timeout-policy idle 60 life 86400 requests 10000

ip http path flash:ccmegui

!

ip dns server

ip nat inside source list NAT interface Dialer14 overload

ip nat inside source list NAT_2 interface Dialer16 overload

ip route 0.0.0.0 0.0.0.0 B.B.160.239

!

ip access-list extended NAT

deny ip 192.168.10.0 0.0.0.255 192.168.11.0 0.0.0.255

deny ip 192.168.10.0 0.0.0.255 192.168.1.0 0.0.0.255

permit ip 192.168.10.0 0.0.0.255 any

ip access-list extended NAT_2

permit ip 192.168.17.0 0.0.0.255 any

ip access-list extended VPN

permit ip 192.168.10.0 0.0.0.255 192.168.11.0 0.0.0.255 log

permit ip 192.168.10.0 0.0.0.255 192.168.1.0 0.0.0.255 log

!

access-list 1 permit 192.168.11.0 0.0.0.255 log

access-list 1 permit 192.168.10.0 0.0.0.255

access-list 2 permit A.A.83.246

access-list 105 permit ip 192.168.10.0 0.0.0.255 192.168.1.0 0.0.0.255

access-list 105 remark SITE-2-SITE ACL

access-list 2000 permit ip host A.A.83.246 host X.X.117.177

access-list 2000 permit ip host A.A.83.202 host X.X.117.177

access-list 2000 permit ip host A.A.81.249 host X.X.117.177

!

route-map VOICE_INTERNET permit 10

match ip address NAT_2

set default interface Dialer16

!

route-map USER_INTERNET permit 4

match ip address VPN

set default interface Dialer12

!

route-map USER_INTERNET permit 5

match ip address 1

set default interface Dialer14

!

control-plane

!

webvpn gateway SSL_GATEWAY

ip address A.A.83.246 port 443

http-redirect port 80

ssl trustpoint TP-self-signed-3690820435

inservice

!

webvpn install svc flash:/webvpn/anyconnect-win-3.1.02040-k9.pkg sequence 1

!

webvpn context ANYCONNECT_SERVIC

secondary-color white

title-color #CCCC66

text-color black

ssl authenticate verify all

!

policy group policy_1

functions svc-enabled

timeout idle 300

svc address-pool "ANYCONNECT_POOL" netmask 255.255.255.0

svc keep-client-installed

svc split include 192.168.17.0 255.255.255.0

svc split include acl 1

virtual-template 1

default-group-policy policy_1

aaa authentication list ciscocp_vpn_xauth_ml_1

gateway SSL_GATEWAY

inservice

!

end

All interfaces are UP:

ping B.B.160.239 so di 12

ping B.B.160.239 so di 14

ping B.B.160.239 so di 16

C3825#s

Interface IP-Address OK? Method Status Protocol

GigabitEthernet0/0 unassigned YES NVRAM up up

GigabitEthernet0/0.12 unassigned YES unset up up

GigabitEthernet0/0.14 unassigned YES unset up up

GigabitEthernet0/0.16 unassigned YES unset up up

GigabitEthernet0/1 unassigned YES NVRAM administratively down down

FastEthernet0/0/0 unassigned YES unset up up

FastEthernet0/0/1 unassigned YES unset up down

FastEthernet0/0/2 unassigned YES unset up down

FastEthernet0/0/3 unassigned YES unset up down

Dialer12 A.A.83.246 YES IPCP up up

Dialer14 A.A.83.202 YES IPCP up up

Dialer16 A.A.81.249 YES IPCP up up

NVI0 unassigned YES unset administratively down down

Virtual-Access1 unassigned YES unset down down

Virtual-Access2 A.A.83.246 YES unset up up

Virtual-Access3 unassigned YES unset up up

Virtual-Access4 unassigned YES unset up up

Virtual-Access5 unassigned YES unset up up

Virtual-Access6 unassigned YES unset up up

Virtual-Template1 A.A.83.246 YES unset down down

Vlan1 192.168.10.162 YES NVRAM up up

Vlan17 192.168.17.1 YES NVRAM up up

C3825#ping B.B.160.239 so di 14

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to B.B.160.239, timeout is 2 seconds:

Packet sent with a source address of A.A.83.202

....

Success rate is 0 percent (0/4)

C3825#ping B.B.160.239 so di 12

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to B.B.160.239, timeout is 2 seconds:

Packet sent with a source address of A.A.83.246

....

Success rate is 0 percent (0/4)

C3825#ping B.B.160.239 so di 16

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to B.B.160.239, timeout is 2 seconds:

Packet sent with a source address of A.A.81.249

....

Success rate is 0 percent (0/4)

C3825#

Interface Dialer12 & Dialer16 are DOWN:

C3825#s

Interface IP-Address OK? Method Status Protocol

GigabitEthernet0/0 unassigned YES NVRAM up up

GigabitEthernet0/0.12 unassigned YES unset up up

GigabitEthernet0/0.14 unassigned YES unset up up

GigabitEthernet0/0.16 unassigned YES unset up up

GigabitEthernet0/1 unassigned YES NVRAM administratively down down

FastEthernet0/0/0 unassigned YES unset up up

FastEthernet0/0/1 unassigned YES unset up down

FastEthernet0/0/2 unassigned YES unset up down

FastEthernet0/0/3 unassigned YES unset up down

Dialer12 unassigned YES IPCP administratively down down

Dialer14 A.A.83.202 YES IPCP up up

Dialer16 unassigned YES IPCP administratively down down

NVI0 unassigned YES unset administratively down down

Virtual-Access1 unassigned YES unset down down

Virtual-Access2 unassigned NO unset up up

Virtual-Access3 unassigned YES unset up up

Virtual-Access4 unassigned YES unset down down

Virtual-Access5 unassigned YES unset up up

Virtual-Access6 unassigned YES unset down down

Virtual-Template1 unassigned NO unset down down

Vlan1 192.168.10.162 YES NVRAM up up

Vlan17 192.168.17.1 YES NVRAM up up

C3825#ping B.B.160.239 so di 14

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to B.B.160.239, timeout is 2 seconds:

Packet sent with a source address of A.A.83.202

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

C3825#

antonio.guirado · ‎05-08-2013

Hello Alexandr,

I have studied you configuration and all posts. Finally, I think you are rigth. The root cause for ICMP traffic genereated from your router does not work fine is balancing. Router receives ICMP Request in on dialer and the ICMP Reply goes back in other Dialer. You ISP applies anti-spoofing filters and drops the packet. Notice that this is only for traffic generated for the router.

You can play with administrative distances in static routes. For example, if you want for local traffic only used

Dialer 14 you can use:

no ip route 0.0.0.0 0.0.0.0 B.B.160.239

ip route 0.0.0.0 0.0.0.0 Dialer 12 100

ip route 0.0.0.0 0.0.0.0 Dialer 14 50

ip route 0.0.0.0 0.0.0.0 Dialer 16 150

If Dialer 12 is only for a site-to-site VPN you can route only the remote ip address:

ip route VPN-peer-address 255.255.255.255 Dialer 12

Traffic coming from ethernets interfaces is applied a PBR and forced to go through a dialer. In this case, NAT

translates the source inside address and the reply packet should come in the right dialer. Do you have any issues

with users traffic?. Is it right?.

Regards

Alexandr Matskevich · ‎05-09-2013

Hello Antonio,

no ip route 0.0.0.0 0.0.0.0 B.B.160.239

ip route 0.0.0.0 0.0.0.0 Dialer 12 100

ip route 0.0.0.0 0.0.0.0 Dialer 14 50

ip route 0.0.0.0 0.0.0.0 Dialer 16 150

"100", "50" and "150" in these strings are not administrative distance. This is metric's value. I tried to change metric from default and 2 vlans goes to Internet through different Dialer interface in case of Dialer14 and Dialer16 are "up" and in the same time Dialer 12 is down.But ICMP-reply still not returned.

I want to have some public servers (mail-server, and Surveillance) available from Internet through different Dialer interfaces and session started from Internet will not return (like ICMP-request). In my opinion, construction with different metrics is not proper solution. Now I'm trying to split my router using VRF technology. In theory VRF may help.

Sorry for my terrible English.

antonio.guirado · ‎05-09-2013

Hello Alexandr,

review ip route command because is administrative distance and not metric. Perhaps the name "distance metric"

is a little confusing in the ip route command but its value it in the range 1-255.

At the end, I can know the real problem. You need access from internet to your servers that have private address,

don't you?. VRF Lite can let you use different routing table and its a very powerful feature. Notice that the routing

tables are differents, so if you want routing between them you must configure it with static routes from one VRF to other.

But I think it is not your problem. The problem is that you must to configure inverse NAT to map the public address to the server private address. For example, to map to a mail server:

ip nat inside source static tcp 25 25 extendable

Be careful that others NAT command (with the overload keywords) does not include the private_address.

Good luck!!

Regards