Solved: What does ICMP message (11,1) mean?

jgtheodor · ‎12-03-2009

Hi,

I am working in a DMVPN environment with two HUB and 25 Spoke routers. There are mGRE tunnels everywhere with the same basic configuration. There are also attached in WAN Serial & ADSL interfaces Extended Access Lists permitting only the esp and ISAKMP (udp 500) packets. Every day in the Primary HUB router I see the following log messages:

Dec 03 08:52:57 172.16.250.2 2528762: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.30 -> 192.168.192.1 (11/1), 13 packets
Dec 03 08:52:57 172.16.250.2 2528763: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.26 -> 192.168.192.1 (11/1), 8 packets
Dec 03 08:52:57 172.16.250.2 2528764: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.82 -> 192.168.192.1 (11/1), 1 packet
Dec 03 08:53:57 172.16.250.2 2528765: Dec 3 08:53:44.148: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.78 -> 192.168.192.1 (11/1), 8 packets

The source IP Addresses are the WAN IP addresses of all Spoke routers and the IP address 192.168.192.1 is the Loopback IP address of Primary HUB router. Similar log messages I see in every Spoke router, with source IP Address the Primary HUB WAN Interface and destination IP Addresses the Loopback IP Addresses of all other Spoke routers. As far I know there is no any fragmentation issue, and everything works fine. But the answer remains:

Where these ICMP packets come from?

Can anyone help me answer this question?

Thanks in advance!

Michael Sullenberger · ‎12-18-2009

I got my reply from the developers, and they pointed out an error that I had

in my overhead byte calculation.

So with GRE(tunnel_key)+AH+ESP(AES)+ESPauth IPsec transport mode,

a 1400 byte clear text packet will end up being 1504 bytes after encapsulation

and encryption. So you definitely need to lower your IP MTU. Again I would

recommend 'ip mtu 1380' and 'ip tcp adjust-mss 1340'. A 1380 byte clear text

packet will end up being 1472 bytes (encapsuleated+encrypted). This will

give you a little extra room in case there are other overheads, like NAT-T

(8 bytes) or PPP-E (8 bytes).

Note, I was thinking about your dialer interface. I think you want to use

'ip mtu 1492' and 'ip tcp adjust-mss 1452'. I think there is only 8 bytes

of overhead, unless your Dialer is for some reason using more.

Mike.

View solution in original post

Giuseppe Larosa · ‎12-03-2009

Hello John,

message type 11:

11     Time Exceeded                     [RFC792]

1     Unassigned                                        [JBP]

see

http://www.iana.org/assignments/icmp-parameters

Hope to help
Giuseppe

jgtheodor · ‎12-03-2009

Hi Giuseppe,

I know what does this ICMP message mean, but actually I am looking to find out where these messages come from and why they appear in my network...

I have already inspected this issue with my ISP Cisco engineers and they do not find any misconfiguration or something like that. It seems that something triggers the routers to produce these ICMP packets but I do not know what!

However thanks for your response.

Giuseppe Larosa · ‎12-04-2009

Hello John,

a good starting point would be to post the named ACL WAN that is denying these packets and the configuration to see if those IP addresses are related to external public ip addresses or ip addresses on multipoint GRE.

also another title for the thread would have been appropriate: I have read too quickly your original post but the title is misleading.

Edit:

>> Where these ICMP packets come from?

they are apparentently sourced by first ip address that appears in the log messages X-> Y, check those devices if they are under your control.

it might be a result that these devices belong to somebody else

Hope to help

Giuseppe

jgtheodor · ‎12-04-2009

You are right Giuseppe,

The title is misleading. Sorry for that honestly.

Now, regarding these IP Addresses: The first one (X) represent the WAN IP Addresses of Spoke Routers and the second one (Y) represent the Loopback IP Address of primary HUB router. These devices are under of my control. It seems that for some reason the spoke routers send these ICMP messages to HUB router. The source IP address is the WAN IP Address Serila Interface and the destination IP Address is the HUB router Loopback IP Address where the VPN IPSec Tunnels are established.

You can see below the Extended ACL that is applied in all WAN interface in all routers in VPN cloud. The network 192.168.192.0 represent the Loopback IP Addresses we use for the IPSec end-points. For example branch A has the Loopback 0 IP Address 192.168.192.10, Branch B 192.168.192.11 etc and finally HUB has 192.168.192.1.

ip access-list extended WAN
permit ahp 192.168.192.0 0.0.15.255 192.168.192.0 0.0.15.255
permit udp 192.168.192.0 0.0.15.255 eq isakmp 192.168.192.0 0.0.15.255 eq isakmp
deny ip any any log

Giuseppe Larosa · ‎12-04-2009

Hello John,

no problems

>> It seems that for some reason the spoke routers send these ICMP messages to HUB router.

I wonder if there are configured IP SLAs on spoke routers that use ICMP probes.

Hope to help

Giuseppe

jgtheodor · ‎12-04-2009

No there are not configured IP SLAs in either HUB & Spoke routers. But as far I know there must be configured in PE routers because our ISP wants to monitor and track the state of WAN links. Do you think that this can cause the relevant ICMP messages? If the answer is yes, why the source IP Address is the WAN IP Address of my router?

Thanks for your help!!!

marikakis · ‎12-05-2009

I am not very familiar with such scenarios, but have a couple of thoughts on this, which might help.
The ICMP type 11 with code 1 means ICMP Time Exceeded/Fragmentation Reassembly Timeout.
If we ignore the possibility of a bug for a moment, to me this means not only that fragmentation is happening,
but also that the routers are doing the reassembly, and they sometimes timeout when doing it. 
Typically the hosts do the reassembly and not the routers, but cases of tunneling are an exception.
When you use IPsec for example, reassembly might be needed at the router so that packet can be decrypted.
Also, the IP addresses in the logs make me think this issue has to do with tunnels between the routers.
Generally the generation of an icmp time-exceeded message might be triggered remotely (e.g. traceroute 
program), but the destination of the icmp time-exceeded typically points to the original device that 
triggered it (of course the traffic traversing the tunnel can still be remote). 
Please take a look at the following document:
http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml#t16
What are the MTU settings of your network? How often do those messages appear? 
Are you monitoring the CPU of your routers and does anything unusual appear there? 

Edit: Sorry for the format of this message, but something goes wrong while editing and haven't managed to fix it up to now.

jgtheodor · ‎12-06-2009

Hello Maria,

The Tunnels which are mGRE (HUB & Spokes) have been configured with MTU 1400 & MSS 1360. I see these log messages many times during the day, mostly in the HUB router. Almost every 5-10 minutes. I have monitored the CPU load not only for the HUB router (3845) but for all the other spoke routers (2811). Nothing unsual or abnormal except those log messages. The IOS Image in 3845 is 12.4(24)T1 and in 2811 routers 12.4(15)T8.

I will check this document carefully and I hope to find something useful. Keep in mind that I have opened a TAC case in Cicso for this behavior but no answer yet since last month.

Thanks Maria!!!

marikakis · ‎12-06-2009

Hi John,

Almost every 5-10 minutes every day (including weekends) or any hours that your network might normally be somewhat silent? Or is there a period that those messages are seen more often? I am asking this because a very regular interval might point to some network management application doing work. Giuseppe already suggested something similar, but I am thinking that an SNMP-based application (SNMP typically uses UDP), might not react to router mss suggestions and might need an extra step to produce equivalent behavior. Other than that, and for the TCP case, I am thinking about your inbound interfaces leading to the tunnels. If you could afford and you think it's worth even though you do not have any other obvious issues besides those messages (and while you are waiting for cisco to respond without excluding the possibility of some bug), maybe you could try to adjust-mss to some inbound interface towards one of the tunnels to see if this makes any difference. I must say however that all those are just theories, and by no means have I managed to conclude on some specific procedure that is causing this.

Kind Regards,

Maria

jgtheodor · ‎12-06-2009

Maria,

The relevant log messages are seen almost every day (including weekends) in HUB router. They are appeared every 5, 10 or 15 minutes every time. Sometimes maybe 30 minutes, I mean it is not a periodic phenomenon, but only in the HUB router. In the spoke routers the relevant log messages are not seen every day. There are spoke routers on which, these log messages have not been appeared for 6 or in some cases for 10 days. In any case the period in which those messages are appeared in spoke routers - when they are appeared - is 1-3 times per day and general not all days.

I read the document you send me. It is great but I just verified that the Tunnel's configuration is correct. I do not use PMTUD, because I have configured static IP MTU and MSS in Tunnel interfaces. I have also already upgraded the HUB router from 12.4(21)T1 to 12.4(24)T1 and I was thinking to proceed in 15.0(1)M.

I am also thinking that it migth useful to follow your suggestion tommorow and configure MSS in LAN interfaces and monitor to see if this makes difference.

In any case I will update the Discussion.

Thanks again and keep walking!

jgtheodor · ‎12-07-2009

Good day again,

After a thorough investigation of HUB router's log messages of the last 6 days, I realized that Maria was right. When the network is somewhat silent during the weekend the number of relevant ICPM packets were too small. I also realized the those messages start appearing every morning at 7:30 - 8:00 and the number of those ICPM packets/ log messages get decreased after 18:00 or 19:00 (branches end day time). It seems that when there is spoke to HUB traffic these ICMP packets are generated for some reason.

Earlier, this morning I added in 2 Branch router's FastEthernet interface the command "ip tcp adjust-mss 1360" and I am monitoring to see if this change makes the difference.

I am also sending you the relevant log messages of the last 6 days in HUB router.

The IP addresses from network 10.195.35.0 are the IP addresses for CE - PE branches WAN connections. The IP Address 192.168.192.1 is the Loopback IP Address in HUB router where the Tunnels from branches are established. The WAN IP Addresses in HUB router is 10.195.35.6 and 10.195.35.10. Do not confuse from two IP Addresses beacuse the Primary HUB router has 2 Serial interfaces. These two Serial interfaces are connected to PE router and the HUB router perform per packet load-balancing as well as the PE router.

jgtheodor · ‎12-07-2009

I would like to update the Discusion with the results from the "ip tcp adjust-mss 1360" command adding in 2 branch router's LAN interfaces:

Nothing at all. I keep taking the same ICMP packets (11,1). I have used the command "debug ip icmp" without any obvious result.

marikakis · ‎12-07-2009

Hi John,

I didn't have a chance to look very closely at the times the ACL messages occur, because you have other messages as well in all the logs you posted, including a severity 1 error. For example:

Dec 05 12:50:04 172.16.250.2 2530095: Dec 5 12:50:03.459: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.192.33,dstadr=192.168.192.1,size=1504,handle=0x6A1B
Dec 05 12:50:05 172.16.250.2 2530096: Dec 5 12:50:03.459: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=4635 local=192.168.192.1 remote=192.168.192.33 spi=FCF866BB seqno=00000061

Please see the following document:

https://supportforums.cisco.com/docs/DOC-2193

Kind Regards,

Maria

jgtheodor · ‎12-07-2009

Hello Maria,

I have opened a TAC case in Cisco for those error log messages two months ago and the TAC Engineer told me that these log messages are just a cosmetic bug and nothing more. Of course the relevant log message does not disrupt the VPN traffic, but If I understand, there are fragment packets in the network which generate the relevant ICMP packets as well as VPN error log messages.

I will try further investigation and update accordingly the Conversation.

Thank you for the great information!