Duplicate multicast packets with DMVPN tunnel

[nicblais1]

I have a setup where a spoke (cisco 1841) is sending a multicast feed to a hub (cisco 2951) via a DMVPN tunnel on the Internet. The feed arrives on interface fa0/0 of the cisco 1841 and is forwarded to the tunnel interface.  It is about 160,000 kbit/s and 18 pps. This always looks the same:

cisco2951-1-hub#sh run int tu10

!

interface Tunnel10

description DMVPN TUNNEL

bandwidth 2000

ip address 192.168.200.1 255.255.255.0

no ip redirects

ip mtu 1400

ip pim dr-priority 100

ip pim sparse-mode

no ip next-hop-self eigrp 1

ip nhrp map multicast dynamic

ip nhrp network-id 200

ip tcp adjust-mss 1360

no ip split-horizon eigrp 1

load-interval 30

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 200

tunnel protection ipsec profile DMVPN

end

cisco1841-1-spoke#sh int fa0/0  //FEED SOURCE

FastEthernet0/0 is up, line protocol is up

  Hardware is Gt96k FE, address is 0026.0b25.3294 (bia 0026.0b25.3294)

  Description: INTERFACE TO INTERNET

  Internet address is 192.168.1.135/24

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 163000 bits/sec, 18 packets/sec

     1200 packets input, 180041 bytes

     Received 264 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog

     0 input packets with dribble condition detected

     23371 packets output, 27131074 bytes, 0 underruns

     0 output errors, 0 collisions, 2 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

In the following spoke tunnel config for DMVPN (the tunnel on the spoke have a tunnel destination set), the feed is forwarded properly:

cisco1841-1-spoke#sh run int tu10

interface Tunnel10

description DMVPN TUNNEL

bandwidth 2000

ip address 192.168.200.3 255.255.255.0

ip mtu 1400

ip pim sparse-mode

ip nhrp map multicast public.ip

ip nhrp map 192.168.200.1 public.ip

ip nhrp network-id 200

ip nhrp nhs 192.168.200.1

ip tcp adjust-mss 1360

load-interval 30

tunnel source FastEthernet0/0

tunnel destination public.ip

tunnel key 200

tunnel protection ipsec profile DMVPN

end

cisco1841-1-spoke#sh int tu10

Tunnel10 is up, line protocol is up

  Hardware is Tunnel

  Description: DMVPN TUNNEL

  Internet address is 192.168.200.3/24

  MTU 17912 bytes, BW 2000 Kbit/sec, DLY 50000 usec,

     reliability 255/255, txload 20/255, rxload 1/255

  Encapsulation TUNNEL, loopback not set

  Keepalive not set

  Tunnel source 192.168.1.135 (FastEthernet0/0), destination public.ip

   Tunnel Subblocks:

      src-track:

         Tunnel10 source tracking subblock associated with FastEthernet0/0

          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>

  Tunnel protocol/transport GRE/IP

    Key 0xC8, sequencing disabled

    Checksumming of packets disabled

  Tunnel TTL 255, Fast tunneling enabled

  Tunnel transport MTU 1472 bytes

  Tunnel transmit bandwidth 8000 (kbps)

  Tunnel receive bandwidth 8000 (kbps)

  Tunnel protection via IPSec (profile "DMVPN")

  Last input 00:00:02, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 5

  Queueing strategy: fifo

  Output queue: 0/0 (size/max)

  30 second input rate 0 bits/sec, 0 packets/sec

  30 second output rate 174000 bits/sec, 18 packets/sec

     463 packets input, 39469 bytes, 0 no buffer

     Received 0 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

     23466 packets output, 26148514 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

but here is my problem... When I actually make the tunnel fully mesh as I want to make it:

cisco1841-1-spoke#sh run int tu10

interface Tunnel10

description DMVPN TUNNEL

bandwidth 2000

ip address 192.168.200.3 255.255.255.0

no ip redirects

ip mtu 1400

ip pim sparse-mode

ip nhrp map multicast public.ip

ip nhrp map 192.168.200.1 public.ip

ip nhrp network-id 200

ip nhrp nhs 192.168.200.1

ip tcp adjust-mss 1360

load-interval 30

tunnel source FastEthernet0/0

tunnel mode gre multipoint

tunnel key 200

tunnel protection ipsec profile DMVPN

end

I get duplicate packets:

cisco1841-1-spoke#sh int tu10   

Tunnel10 is up, line protocol is up

  Hardware is Tunnel

  Description: DMVPN TUNNEL

  Internet address is 192.168.200.3/24

  MTU 17912 bytes, BW 2000 Kbit/sec, DLY 50000 usec,

     reliability 255/255, txload 42/255, rxload 1/255

  Encapsulation TUNNEL, loopback not set

  Keepalive not set

  Tunnel source 192.168.1.135 (FastEthernet0/0)

   Tunnel Subblocks:

      src-track:

         Tunnel10 source tracking subblock associated with FastEthernet0/0

          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface <OK>

  Tunnel protocol/transport multi-GRE/IP

    Key 0xC8, sequencing disabled

    Checksumming of packets disabled

  Tunnel TTL 255, Fast tunneling enabled

  Tunnel transport MTU 1472 bytes

  Tunnel transmit bandwidth 8000 (kbps)

  Tunnel receive bandwidth 8000 (kbps)

  Tunnel protection via IPSec (profile "DMVPN")

  Last input 00:00:00, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 10

  Queueing strategy: fifo

  Output queue: 0/0 (size/max)

  30 second input rate 0 bits/sec, 0 packets/sec

  30 second output rate 330000 bits/sec, 36 packets/sec

     574 packets input, 49186 bytes, 0 no buffer

     Received 0 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

     31866 packets output, 35490190 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

The feed is also unusable.

Even mstat reports it:

cisco2951-1-hub#mstat 192.168.103.100 192.168.200.1 239.255.0.47

Type escape sequence to abort.

Mtrace from 192.168.103.100 to 192.168.200.1 via group 239.255.0.47

From source (?) to destination (?)

Waiting to accumulate statistics......

Results after 10 seconds:

  Source        Response Dest   Packet Statistics For     Only For Traffic

192.168.103.100 192.168.200.1   All Multicast Traffic     From 192.168.103.100

     |       __/  rtt 27   ms   Lost/Sent = Pct  Rate     To 239.255.0.47

     v      /     hop 27   ms   ---------------------     --------------------

192.168.103.1  

192.168.200.3   ?

     |     ^      ttl   0  

     v     |      hop 0    ms    -183/181 = --%      18 pps    0/0 = --%  0 pps

192.168.200.1   ? Reached RP/Core

     |      \__   ttl   1  

     v         \  hop 0    ms        0         0 pps           0    0 pps

192.168.200.1   192.168.200.1  

  Receiver      Query Source

Really have no idea why this is doing this... I'd like to have my setup take advantage of the dynamic tunnels vs spoke-to-hub only... Any suggestions?

[Peter Paluch]

Hello,

I am also at a loss why should a simple change from a  point-to-point to multipoint GRE configuration on a tunnel lead to  doubling the traffic. Are you absolutely sure you do not have any NHRP  mapping on the spoke router present with the multicast flag except the  hub?

One thing, though. The DMVPN behaves like a NBMA  network to its participants. Therefore, all routers in DMVPN should be  using PIM NBMA mode on their multipoint GRE tunnels. This is  accomplished by using ip pim nbma-mode command on the Tunnel interfaces. Read more about the need for NBMA mode here:

http://www.cisco.com/en/US/docs/ios/solutions_docs/ip_multicast/White_papers/frm_rlay.html

http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/DMVPN_2_Phase2.html#wp132557

Would  the doubled traffic problem still persist even after you configure the  PIM NBMA mode on tunnel interfaces? By the way, where is your PIM RP  router topologically located?

Best regards,

Peter

[nicblais1]

Hi Peter,

Thank you for answering.  I have no other NHRP mapping than what you see above.  The rp-address is actually the hub router (192.168.200.1).  I've also tried setting it to be the loopback address on the hub but it doesn't change the behavior.

I have set ip pim nbma-mode on each router's tunnel interface, and believe it or not, I get triple the packet:

cisco1841-1-spoke#sh int tu10

Tunnel10 is up, line protocol is up

  Hardware is Tunnel

  Description: DMVPN TUNNEL

  Internet address is 192.168.200.3/24

  MTU 17912 bytes, BW 2000 Kbit/sec, DLY 50000 usec,

     reliability 255/255, txload 63/255, rxload 1/255

  Encapsulation TUNNEL, loopback not set

  Keepalive not set

  Tunnel source 192.168.1.137 (FastEthernet0/0)

   Tunnel Subblocks:

      src-track:

         Tunnel10 source tracking subblock associated with FastEthernet0/0

          Set of tunnels with source FastEthernet0/0, 1 member (includes iterators), on interface

  Tunnel protocol/transport multi-GRE/IP

    Key 0xC8, sequencing disabled

    Checksumming of packets disabled

  Tunnel TTL 255, Fast tunneling enabled

  Tunnel transport MTU 1472 bytes

  Tunnel transmit bandwidth 8000 (kbps)

  Tunnel receive bandwidth 8000 (kbps)

  Tunnel protection via IPSec (profile "DMVPN")

  Last input 00:00:00, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 38

  Queueing strategy: fifo

  Output queue: 0/0 (size/max)

  30 second input rate 0 bits/sec, 0 packets/sec

  30 second output rate 497000 bits/sec, 55 packets/sec

     12520 packets input, 1041053 bytes, 0 no buffer

     Received 0 broadcasts (1 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

     690352 packets output, 770591782 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

Once I remove ip pim nbma-mode on the spoke but leave it on the hub, it goes back to being duplicates...  I also tried disabling cef but nothing changed...

[Peter Paluch · Accepted solution]

Hello,

This is curious indeed. The fact is, though, that you are required to run PIM NBMA mode - I believe you understand the reasons described in the documents I have referenced earlier. Also, you should not deactivate IP CEF - process switching without IP CEF could bring your router down to its knees.

Can you please post the show ip mroute from both the spoke and the hub during the multicast delivery when the traffic on the tunnel is doubled or tripled? Thank you!

Best regards,

Peter

[nicblais1]

Peter,

Good news!  I added ip pim nbma-mode back on all routers, and reloaded all of them, and it worked.  It probably didn't work the first time when I added it as the sessions were still up after the change and reloading each spokes and the hub did it.  Now all is good!

Many thanks!

[Peter Paluch]

Hello,

Glad I could be of help! And, oh, regarding the solution by restarting the devices... Sometimes the easiest approach just works

Best regards,

Peter