Solved: Re: What does ICMP message (11,1) mean? - Page 3

jgtheodor · ‎12-03-2009

Hi,

I am working in a DMVPN environment with two HUB and 25 Spoke routers. There are mGRE tunnels everywhere with the same basic configuration. There are also attached in WAN Serial & ADSL interfaces Extended Access Lists permitting only the esp and ISAKMP (udp 500) packets. Every day in the Primary HUB router I see the following log messages:

Dec 03 08:52:57 172.16.250.2 2528762: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.30 -> 192.168.192.1 (11/1), 13 packets
Dec 03 08:52:57 172.16.250.2 2528763: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.26 -> 192.168.192.1 (11/1), 8 packets
Dec 03 08:52:57 172.16.250.2 2528764: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.82 -> 192.168.192.1 (11/1), 1 packet
Dec 03 08:53:57 172.16.250.2 2528765: Dec 3 08:53:44.148: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.78 -> 192.168.192.1 (11/1), 8 packets

The source IP Addresses are the WAN IP addresses of all Spoke routers and the IP address 192.168.192.1 is the Loopback IP address of Primary HUB router. Similar log messages I see in every Spoke router, with source IP Address the Primary HUB WAN Interface and destination IP Addresses the Loopback IP Addresses of all other Spoke routers. As far I know there is no any fragmentation issue, and everything works fine. But the answer remains:

Where these ICMP packets come from?

Can anyone help me answer this question?

Thanks in advance!

marikakis · ‎12-16-2009

Did you do the math and came up with 1320? Because I did some fast and dirty calculations and came up with the same value!

The payload of 1372 of ICMP message (user data) is encapsulated with 20 bytes IP and 8 bytes ICMP headers for a total of 28 bytes.

http://en.wikipedia.org/wiki/Ping#ICMP_packet

The payload of TCP segment is encapsulated with 20+60 = 80 bytes maximum

http://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_segment_structure

"The minimum size header is 5 words and the maximum is 15 words thus giving the minimum size of 20 bytes and maximum of 60 bytes, allowing for up to 40 bytes of options in the header."

80 - 28 = 52 bytes difference between ICMP and TCP, so 1372 - 52 = 1320 bytes!

I did this really quickly and might not be perfectly correct. In any case, the point is to try to tune the MSS to your environment.

jgtheodor · ‎12-16-2009

Yeap Mary,

I think so... I will try one of the next 3-4 days to tune the MSS size in LAN interfaces at 1300 for sure and monitor. It seems that the MSS makes the difference...

Anyway, thanks again for your assistance!

I will keep you informed

Phillip Remaker · ‎12-16-2009

You probably have a routing loop somewhere.

Are you using BGP? External BGP by default uses a TTL of 1, so external BGP packets may be discarded with that error if they try to go more than one hop.

DMVPN routing can be tricky, especially when the hub address is on the same subnet as the tunneled address ranges.

Ignore previous message. It was completely wrong.

ICMP 11/1 is "Fragmentation Reassembly Time Exceeded." Not "Time Exceeded."

This is more consistent with your other observations. Looks like for longer packets, the first fragment makes it but not the second fragment, probably due to some access list issue.

jgtheodor · ‎12-16-2009

No, I do not use BGP at all. EIGRP is the only enabled routing protocol in the whole network. HUB IP Addresses are not on the same subnet as the tunneled address ranges.

Michael Sullenberger · ‎12-17-2009

I believe that the problem you are having is that you are fragmenting GRE/IPsec packets and sometime (or most times) both fragments don't make it to the other side (tunnel end point) and IP reassembly times out and the ICMP 11/1 message is sent to the source of the packet (tunnel end point). In your case I see that you are using AH as well as ESP (possible also ESPauth). The recommended GRE tunnel 'ip mtu 1400' and 'ip tcp adjust-mss 1360' values are based on only using ESP and ESPauth. Note, ESPauth adds 12 bytes of overhead and AH adds 24 bytes of over head. Also AES encryption adds between 8-16 bytes more overhead as compared with 3DES (DES). For a quick test try lowering the 'ip mtu ...' and 'ip tcp adjust-mss ...' values to 1360 and 1340 respectively. We can work out more accurate values later.

PS. I just looked further down in the message stream and saw that you are already going to try reducing these values.

Hopefully the information here will help by telling you why your current values are not low enough.

marikakis · ‎12-17-2009

Hello people from cisco,

It's good to see you coming to the rescue. To cut a long story short, John has opened cases for months now and is waiting for official answer from cisco to proceed (officially) with the changes. I am not very familiar with such setups as I said from the beginning, but could feel his pain. If you can pull some strings so he can get a definitive answer, it would be nice. I am not saying that you must. I am only saying it would be nice.

Kind Regards,

Maria

jgtheodor · ‎12-17-2009

Good day,

Well mls, thank you for your detailed information. I have configured in HUB and spoke routers the following crypto commands:

******************************************************************************************************

crypto isakmp policy 10
encr aes 256
authentication pre-share
group 5
crypto isakmp key ######### address 0.0.0.0 0.0.0.0
crypto isakmp keepalive 10
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set IPSEC-AES-256 ah-sha-hmac esp-aes 256 esp-sha-hmac
mode transport
!
crypto ipsec profile PROFILE
set transform-set IPSEC-AES-256
set pfs group2

******************************************************************************************************

Now, is it possible from this configuration to compute how much is the extra accurate overhead in every IP packet traversing the GRE/IPSec Tunnel? Keep in mind the Ping Test results in a previous post.

Now for the time being, I have configured the MSS value at 1300 in HUB's LAN & Tunnel interfaces since Yesterday afternoon and there is no any ICMP message (11,1) till now . Mary had right, It seems tha the MMS value makes the difference! However in all spoke routers the MSS value remains 1360. Could you please inform me if I must also change the MTU along with the MSS, or the MTU can remains at 1400? Do you believe that the relevant MSS & MTU must be configured only in Tunnel interfaces, or in LAN interfaces?

I just want only an accurate MSS value to proceed accordingly!

PS1. Do you believe that I could change the crypto configuration and use only AH or ESP lowering the extra overhead to increase in this way the performance (all routers do hardware encryption)?

PS2. Mary I have no words. Your assistance was very important. Keep walking!!!

marikakis · ‎12-17-2009

Hi John,

Good news! What about the other messages (hardware errors and routing protocol)? I need this information before starting to celebrate!

Kind Regards,

Maria

jgtheodor · ‎12-18-2009

Well Mary,

Till now, I have no strange EIGRP Adjacency Loss but I have the logs below:

Dec 18 04:52:27 172.16.250.2 2536506: Dec 18 04:52:09.574: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.192.30,dstadr=192.168.192.1,size=144,handle=0x689F
Dec 18 04:52:28 172.16.250.2 2536507: Dec 18 04:52:09.574: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=4255 local=192.168.192.1 remote=192.168.192.30 spi=1AA2D61F seqno=0000002C
Dec 18 05:31:25 172.16.250.2 2536508: Dec 18 05:31:07.603: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.192.33,dstadr=192.168.192.1,size=1136,handle=0x5A35
Dec 18 05:31:26 172.16.250.2 2536509: Dec 18 05:31:07.603: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=565 local=192.168.192.1 remote=192.168.192.33 spi=CEA7987E seqno=0000004B
Dec 18 07:26:39 172.16.250.2 2536510: Dec 18 07:26:22.144: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.200.33,dstadr=192.168.200.1,size=208,handle=0x646F
Dec 18 07:26:40 172.16.250.2 2536511: Dec 18 07:26:22.144: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=3183 local=192.168.200.1 remote=192.168.200.33 spi=085E6012 seqno=0000000F
Dec 18 07:31:44 172.16.250.2 2536512: Dec 18 07:31:27.048: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.192.30,dstadr=192.168.192.1,size=336,handle=0x666B
Dec 18 07:31:45 172.16.250.2 2536513: Dec 18 07:31:27.048: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=3691 local=192.168.192.1 remote=192.168.192.30 spi=240D0DC9 seqno=0000015E

Dec 18 08:06:27 172.16.250.2 2536518: Dec 18 08:06:10.665: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.192.30,dstadr=192.168.192.1,size=608,handle=0x7173
Dec 18 08:06:28 172.16.250.2 2536519: Dec 18 08:06:10.665: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=6515 local=192.168.192.1 remote=192.168.192.30 spi=72226C68 seqno=0000012C
Dec 18 08:17:27 172.16.250.2 2536520: Dec 18 08:17:10.751: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.192.30,dstadr=192.168.192.1,size=144,handle=0x59EF
Dec 18 08:17:28 172.16.250.2 2536521: Dec 18 08:17:10.751: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=495 local=192.168.192.1 remote=192.168.192.30 spi=86A07E4D seqno=00000075

I believe that if I will change the MSS value in the spoke routers as well, I will get a better picture.

Anyway, in any case I would like to thank you again for your assistance! It was a suprise for me to meet a Greek woman in this forum , since I met the first CCIE women in Greece 8 years ago.

I will try more accurate values for MSS the next days and I hope to fix the problem. I will definitely update the Conversation!

marikakis · ‎12-18-2009

FYI: In the ISP I was working a woman there (my mentor) still rules the core. Our boss probably liked that we hardly ever made a mess, and even when we did, we knew how to clean up nicely! I don't know if you had a chance to follow this event: https://supportforums.cisco.com/thread/344391.

"Matt has been with Cisco since starting as a college intern in 1996. His major qualification continues to be an unnatural ability to rack-mount large routers unassisted". We both still don't possess Matt's skills however!

Michael Sullenberger · ‎12-18-2009

There are a number of issues with your setup.

With your setup; AH, ESP AES, ESPauth, IPsec transport mode and GRE
with tunnel key my calculation of the exact size of an encrypted 1400 data/IP
packet would be 1500 bytes. BUT, when I tested this on a router the resulting
size is 1504 bytes. I am checking with the developers about the difference.

For now I would go with 'ip mtu 1380' and 'ip tcp adjust-mss 1340'

Note, Only change the IP MTU and TCP MSS on the tunnel interfaces, do
not change these on other interfaces, except in cases where these interfaces
may have their own "overhead" (like next item).
In looking at your open case I noticed that you have a Dialer1 interface that
has 'ip mtu 1452' configured. If this interface is ever used for outgoing DMVPN
packets then you will need to lower the IP mtu and TCP MSS on the tunnel
interfaces. In this case I would use 'ip mtu 1340' and 'ip tcp adjust-mss 1300'.
Setting the IP MTU is the most important number since it will effect all types of
IP packets. TCP MSS will only effect TCP packets. I recommend configuring
both and also set the TCP MSS value to be exactly 40 bytes less than the IP
MTU value.
It is very uncommon for people to use AH and to use AH and ESPauth together
is overkill. ESPauth will do a hash over the IPsec encrypted packet but not the
IP header. AH will do a hash over the complete IPsec encrypted packet including
the IP header, except for fields that are suppossed to change as the packet is
forwarded. Note, if you ever have any spokes that are behind NAT (their WAN
address gets NATted) then AH will break the connection.
ESP+ESPauth can be done with a single pass through the encryption engine
AH+ESP requires two passes through the encryption engine. I recommend
usingESP + ESPauth. If you have to use AH then just use AH + ESP.
AH+ESP will 12 bytes more overhead than ESP+ESPauth.
I also noticed that you have reduced the IPsec lifetime to 120 seconds. This is
a very bad idea. It will mean that your nodes will be constantly renegotiating
their IPsec SAs. This will significantly increase your CPU utilization, as well
as set you up for mis-matched SAs and possibly other problems. I very highly
recommend that you go back to the default lifetime of 3600 seconds.

Note, I think that there may be some configuration examples that show using
an IPsec lifetime of 120 seconds. These examples are incorrect.
One last item, is that you have configured 'set pfs group2' which will increase
the calculation load on the router, because it must recalculate Diffe-Hillman(DH)
every time you do a IPsec rekey (1 hr default) versus ISAKMP rekey (24 hr default).

Mike.

marikakis · ‎12-18-2009

Hi Mike,

You are the definition of 'extremely helpful'! I wish I had all the necessary knowledge to appreciate more your post. One thing I would like to say is that John is working in the banking sector and maybe has more concerns about security than other environments. As for the 1504 size you mentioned, if you see one of the logs I copy-pasted in an earlier post of mine, you will see this number being in John's logs as well. Other than that, I am speechless!

Best Regards,

Maria

Michael Sullenberger · ‎12-18-2009

I got my reply from the developers, and they pointed out an error that I had

in my overhead byte calculation.

So with GRE(tunnel_key)+AH+ESP(AES)+ESPauth IPsec transport mode,

a 1400 byte clear text packet will end up being 1504 bytes after encapsulation

and encryption. So you definitely need to lower your IP MTU. Again I would

recommend 'ip mtu 1380' and 'ip tcp adjust-mss 1340'. A 1380 byte clear text

packet will end up being 1472 bytes (encapsuleated+encrypted). This will

give you a little extra room in case there are other overheads, like NAT-T

(8 bytes) or PPP-E (8 bytes).

Note, I was thinking about your dialer interface. I think you want to use

'ip mtu 1492' and 'ip tcp adjust-mss 1452'. I think there is only 8 bytes

of overhead, unless your Dialer is for some reason using more.

Mike.

jgtheodor · ‎12-19-2009

God bless you Mike!

Great job!!!!!!!!!!!!!!!!!!!!!!

This is the answer I was looking for, all this time. I was 100% sure that something goes wrong.

Anyway, I will configure all Tunnel interfaces with the recommended values asap.

Mary, I have no words. You make the difference.

Keep walking!

Giuseppe Larosa · ‎12-20-2009

Hello Mike,

very good job you have solved this Greek MTU tragedy !!!

probably this information should be inserted in the Solution reference network design for DMVPN that presents only the most common case where an IP MTU of 1400 bytes fits.

Best Regards

Giuseppe