Hi Olev,

realshiro · ‎12-02-2015

Hi,

I'm running OSPF from HQ to many Branch (via dmvpan) but a lot of Branch ospf state are stuck at " Init " state. I found log as below

***************************Router C (HQ)**********************

show ip traffic interface tunnel 0

Tunnel0 IP-IF statistics :

Rcvd: 1115108263 total, 194233704748 total_bytes

0 format errors, 62 hop count exceeded

0 bad header, 0 no route

0 bad destination, 0 not a router

0 no protocol, 0 truncated

10001 forwarded

40757835 fragments, 5335062 total reassembled

21209255 reassembly timeouts, 0 reassembly failures

0 discards, 1079674892 delivers

Sent: 1143964906 total, 120236787964 total_bytes 0 discards

1143964765 generated, 141 forwarded

1317507 fragmented into, 4488456 fragments, 558263 failed

Mcast: 224212369 received, 18651817400 received bytes

0 sent, 0 sent bytes

Bcast: 6073810 received, 0 sent

-------------------------------------------------------------------------------------------------------------

Oct 7 19:09:29.715: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1780, sending broad/multicast, proto=89

Oct 7 19:09:29.715: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1780, output feature, proto=89, TCP Adjust MSS(54), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

Oct 7 19:09:29.715: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1780, output feature, proto=89, Post-Ingress-NetFlow(72), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

Oct 7 19:09:29.715: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1396, sending fragment

Oct 7 19:09:29.715: IP Fragment, Ident = 7393, fragment offset = 0, proto=89

Oct 7 19:09:29.715: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1396, encapsulation failed

Oct 7 19:09:29.715: IP Fragment, Ident = 7393, fragment offset = 0, proto=89

Oct 7 19:09:43.627: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1780, sending broad/multicast, proto=89

Oct 7 19:09:43.627: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1780, output feature, proto=89, TCP Adjust MSS(54), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

Oct 7 19:09:43.627: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1780, output feature, proto=89, Post-Ingress-NetFlow(72), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE

Oct 7 19:09:43.627: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1396, sending fragment

Oct 7 19:09:43.627: IP Fragment, Ident = 20529, fragment offset = 0, proto=89

Oct 7 19:09:43.627: IP: s=10.45.16.2 (local), d=224.0.0.5 (Tunnel0), len 1396, encapsulation failed

Oct 7 19:09:43.627: IP Fragment, Ident = 20529, fragment offset = 0, proto=89!

***************************BRANCH**************************************************

debug hello from router C

*********************************

.Oct 7 19:47:35 th: IP: s=10.45.16.2 (Tunnel0), d=224.0.0.5, len 1396, input feature

.Oct 7 19:47:35 th: IP Fragment, Ident = 39844, fragment offset = 0, proto=89, CAR(45), rtype 0, forus FALSE, sendself FALSE, mtu 0

.Oct 7 19:47:35 th: IP: s=10.45.16.2 (Tunnel0), d=224.0.0.5, len 1396, rcvd 0

.Oct 7 19:47:35 th: IP Fragment, Ident = 39844, fragment offset = 0, proto=89

.Oct 7 19:47:35 th: IP: recv fragment from 10.45.16.2 offset 0 bytes

.Oct 7 19:47:35 th: pak 84891F30 consumed in input feature , packet consumed, MCI Check(63), rtype 0, forus FALSE, sendself FALSE, mtu 0

.Oct 7 19:47:49 th: IP: s=10.45.16.2 (Tunnel0), d=224.0.0.5, len 1396, input feature

.Oct 7 19:47:49 th: IP Fragment, Ident = 54365, fragment offset = 0, proto=89, CAR(45), rtype 0, forus FALSE, sendself FALSE, mtu 0

.Oct 7 19:47:49 th: IP: s=10.45.16.2 (Tunnel0), d=224.0.0.5, len 1396, rcvd 0

.Oct 7 19:47:49 th: IP Fragment, Ident = 54365, fragment offset = 0, proto=89

.Oct 7 19:47:49 th: IP: recv fragment from 10.45.16.2 offset 0 bytes

.Oct 7 19:47:49 th: pak 84897368 consumed in input feature , packet consumed, MCI Check(63), rtype 0, forus FALSE, sendself FALSE, mtu 0

.Oct 7 19:48:02 th: IP: s=10.45.16.2 (Tunnel0), d=224.0.0.5, len 1396, input feature

.Oct 7 19:48:02 th: IP Fragment, Ident = 64175, fragment offset = 0, proto=89, CAR(45), rtype 0, forus FALSE, sendself

What's wrong ? Please advice.Thank you.

Olev Vallaste · ‎12-02-2015

Hi realshiro,

If you see neghbors in 'INIT' state on HQ-router, it seems that HQ sees hellos from branch, but branch doesnt see hellos from HQ.

Are pings working between the tunnel endpoints?

Second, from debug we see fragmented packets, so I would doing conclusion that something wrong with MTU on the path. can you measure MTU on the path?

realshiro · ‎12-03-2015

Hi Olev Vallaste

Ping is okay. For MTU How to verify along the path .

HQ ------(MPLS service)------Branch

Olev Vallaste · ‎12-03-2015

1. Ok, ping is working.

2. please provide 'sho run interface tunnel <tunnel-number>' from each side.

3. Could you show 'deb ip ospf hello' and 'debug ip ospf adj' from HQ and from the Branch ?

For MTU measuring:

You need to specify packet-size and set df-bit when you pinging tunnel endpoints

realshiro · ‎12-03-2015

i'll share my topology

HQ1a---------- ------------HQ2a

Branch
HQ1b---------- ------------HQ2b

Branch is connect to 4 HQ Router by dmvpn

@HQ1 everything are looking fine

@HQ2 OSPF have a problem , cannot ping between hq-branch so that's mtu issue ?

branch1#ping 10.45.16.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.45.16.2, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

1. Configuration on HQ2 - Branch

------------HQ ------------

!
interface Tunnel0
bandwidth 100000
ip address 10.45.16.2 255.255.254.0
no ip redirects
ip mtu 1400
ip flow ingress
ip nhrp authentication dmvpn
ip nhrp map multicast dynamic
ip nhrp map 10.45.16.1 10.100.253.5
ip nhrp map multicast 10.100.253.5
ip nhrp network-id 10000
ip nhrp holdtime 360
ip nhrp nhs 10.100.253.5
ip tcp adjust-mss 1360
ip ospf network point-to-multipoint
ip ospf hello-interval 15
ip ospf mtu-ignore
ip ospf database-filter all out
ip ospf cost 3
delay 1000
tunnel source Loopback0
tunnel mode gre multipoint
tunnel key 10000
tunnel protection ipsec profile dmvpn
!

---------Branch----------

interface Tunnel0
bandwidth 2048
ip address 10.45.17.222 255.255.254.0
no ip redirects
ip mtu 1400
ip nhrp authentication dmvpn
ip nhrp map multicast 10.100.253.6
ip nhrp map 10.45.16.2 10.100.253.6
ip nhrp map multicast 10.100.253.5
ip nhrp map 10.45.16.1 10.100.253.5
ip nhrp network-id 10000
ip nhrp holdtime 360
ip nhrp nhs 10.45.16.1
ip nhrp nhs 10.45.16.2
rate-limit input access-group 190 512000 96000 192000 conform-action transmit exceed-action drop
rate-limit output access-group 180 512000 96000 192000 conform-action transmit exceed-action drop
ip tcp adjust-mss 1360
ip ospf network point-to-multipoint
ip ospf hello-interval 15
delay 1000
tunnel source 10.2.56.250
tunnel mode gre multipoint
tunnel key 10000

-----------------------------------------------------------------------------------------------

2. Configuration on HQ1 - Branch

-------HQ1---------

!
interface Tunnel1
bandwidth 200000
ip address 10.45.18.1 255.255.254.0
no ip redirects
ip mtu 1400
ip flow ingress
ip nhrp authentication dmvpn
ip nhrp map multicast dynamic
ip nhrp map 10.45.18.2 10.45.254.2
ip nhrp map multicast 10.45.254.2
ip nhrp network-id 10001
ip nhrp holdtime 360
ip nhrp nhs 10.45.254.2
ip tcp adjust-mss 1360
ip ospf network point-to-multipoint
ip ospf cost 1
ip ospf hello-interval 15
ip ospf mtu-ignore
ip ospf database-filter all out
delay 1000
tunnel source Loopback0
tunnel mode gre multipoint
tunnel key 10001
tunnel protection ipsec profile dmvpn
!

---------Branch--------

interface Tunnel1
bandwidth 2048
ip address 10.45.19.222 255.255.254.0
ip access-group 120 out
no ip redirects
ip mtu 1400
ip nhrp authentication dmvpn
ip nhrp map multicast 10.45.254.1
ip nhrp map 10.45.18.1 10.45.254.1
ip nhrp map multicast 10.45.254.2
ip nhrp map 10.45.18.2 10.45.254.2
ip nhrp network-id 10001
ip nhrp holdtime 360
ip nhrp nhs 10.45.18.1
ip nhrp nhs 10.45.18.2
rate-limit input access-group 170 64000 17940 24000 conform-action transmit exceed-action continue
rate-limit input access-group 171 256000 48000 96000 conform-action transmit exceed-action continue
rate-limit input access-group 172 512000 96000 192000 conform-action transmit exceed-action continue
rate-limit input access-group 173 128000 24000 48000 conform-action transmit exceed-action drop
rate-limit input access-group 174 128000 24000 48000 conform-action transmit exceed-action drop
rate-limit input access-group 175 256000 48000 96000 conform-action transmit exceed-action drop
rate-limit output access-group 160 64000 17940 24000 conform-action transmit exceed-action continue
rate-limit output access-group 161 256000 48000 96000 conform-action transmit exceed-action continue
rate-limit output access-group 162 512000 96000 192000 conform-action transmit exceed-action continue
rate-limit output access-group 163 128000 24000 48000 conform-action transmit exceed-action drop
rate-limit output access-group 165 256000 48000 96000 conform-action transmit exceed-action drop
rate-limit output access-group 166 768000 144000 288000 conform-action transmit exceed-action drop
ip tcp adjust-mss 1360
ip ospf network point-to-multipoint
ip ospf hello-interval 15
delay 1000
tunnel source 10.2.56.250
tunnel mode gre multipoint
tunnel key 10001

Olev Vallaste · ‎12-03-2015

>>>

@HQ2 OSPF have a problem , cannot ping between hq-branch so that's mtu issue ?

###

No, I think it is not yet mtu issue in the case. icmp-packets very small (64 - 100 bytes). I think it is connectivity problem over the tunnels.

Is tunnel in UP 'state' ?

Is it connectivity exist between tunnels source and destinations? For example between Loobpack@HQ2 and 10.100.253.5@Branch, and between 10.2.56.250@Branch and 10.100.253.6@HQ2 ?

realshiro · ‎12-08-2015

Hi Olev,

Sorry for my previous information about connectivity between hq-branch.

the correct information is all of branch can ping to HQ

HQ# ping 10.2.54.114 source loo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.54.114, timeout is 2 seconds:
Packet sent with a source address of 10.100.253.5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/10/12 ms

HQ# ping 10.2.54.154 source loo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.54.154, timeout is 2 seconds:
Packet sent with a source address of 10.100.253.5
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/16/16 ms

HQ#sh crypto isa sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status

10.100.253.5 10.2.54.114 QM_IDLE 14515 ACTIVE

10.100.253.5 10.2.54.154 QM_IDLE 14026 ACTIVE

**************HQ*********************

show ip traffic interface tunnel 0

Tunnel0 IP-IF statistics :

Rcvd: 1115108263 total, 194233704748 total_bytes

0 format errors, 62 hop count exceeded

0 bad header, 0 no route

0 bad destination, 0 not a router

0 no protocol, 0 truncated

10001 forwarded

40757835 fragments, 5335062 total reassembled

21209255 reassembly timeouts, 0 reassembly failures

0 discards, 1079674892 delivers

Sent: 1143964906 total, 120236787964 total_bytes 0 discards

1143964765 generated, 141 forwarded

1317507 fragmented into, 4488456 fragments, 558263 failed

Mcast: 224212369 received, 18651817400 received bytes

0 sent, 0 sent bytes

Bcast: 6073810 received, 0 sent

Tunnel0 is up, line protocol is up

Hardware is Tunnel

Internet address is 10.45.16.2/23

MTU 17858 bytes, BW 100000 Kbit/sec, DLY 10000 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation TUNNEL, loopback not set

Keepalive not set

Tunnel source 10.100.253.6 (Loopback0)

Tunnel Subblocks:

src-track:

Tunnel0 source tracking subblock associated with Loopback0

Set of tunnels with source Loopback0, 1 member (includes iterators), on interface <OK>

Tunnel protocol/transport multi-GRE/IP

Key 0x2710, sequencing disabled

Checksumming of packets disabled

Tunnel TTL 255, Fast tunneling enabled

Tunnel transport MTU 1418 bytes

Tunnel transmit bandwidth 8000 (kbps)

Tunnel receive bandwidth 8000 (kbps)

Tunnel protection via IPSec (profile "aycaldmvpnprof")

Last input 00:00:00, output never, output hang never

Last clearing of "show interface" counters 21:34:38

Input queue: 0/500/444/0 (size/max/drops/flushes); Total output drops: 41195

Queueing strategy: fifo

Output queue: 0/0 (size/max)

5 minute input rate 246000 bits/sec, 217 packets/sec

5 minute output rate 573000 bits/sec, 279 packets/sec

15984234 packets input, 3317034043 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

22204916 packets output, 3629279006 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 output buffer failures, 0 output buffers swapped out

Fragmentation failed on OSPF packet