topic Re: What does ICMP message (11,1) mean? in Routing and SD-WAN

What does ICMP message (11,1) mean?

jgtheodor — Mon, 04 Mar 2019 14:52:02 GMT

Hi,

I am working in a DMVPN environment with two HUB and 25 Spoke routers. There are mGRE tunnels everywhere with the same basic configuration. There are also attached in WAN Serial & ADSL interfaces Extended Access Lists permitting only the esp and ISAKMP (udp 500) packets. Every day in the Primary HUB router I see the following log messages:

Dec 03 08:52:57 172.16.250.2 2528762: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.30 -> 192.168.192.1 (11/1), 13 packets
Dec 03 08:52:57 172.16.250.2 2528763: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.26 -> 192.168.192.1 (11/1), 8 packets
Dec 03 08:52:57 172.16.250.2 2528764: Dec 3 08:52:44.143: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.82 -> 192.168.192.1 (11/1), 1 packet
Dec 03 08:53:57 172.16.250.2 2528765: Dec 3 08:53:44.148: %SEC-6-IPACCESSLOGDP: list WAN denied icmp 10.195.35.78 -> 192.168.192.1 (11/1), 8 packets

The source IP Addresses are the WAN IP addresses of all Spoke routers and the IP address 192.168.192.1 is the Loopback IP address of Primary HUB router. Similar log messages I see in every Spoke router, with source IP Address the Primary HUB WAN Interface and destination IP Addresses the Loopback IP Addresses of all other Spoke routers. As far I know there is no any fragmentation issue, and everything works fine. But the answer remains:

Where these ICMP packets come from?

Can anyone help me answer this question?

Thanks in advance!

Re: What does ICMP message (11,1) mean?

Giuseppe Larosa — Thu, 03 Dec 2009 14:56:23 GMT

Hello John,

message type 11:

11     Time Exceeded                     [RFC792]

1     Unassigned                                        [JBP]

see

http://www.iana.org/assignments/icmp-parameters

Hope to help
Giuseppe

Re: What does ICMP message (11,1) mean?

jgtheodor — Fri, 04 Dec 2009 06:35:36 GMT

Hi Giuseppe,

I know what does this ICMP message mean, but actually I am looking to find out where these messages come from and why they appear in my network...

I have already inspected this issue with my ISP Cisco engineers and they do not find any misconfiguration or something like that. It seems that something triggers the routers to produce these ICMP packets but I do not know what!

However thanks for your response.

Re: What does ICMP message (11,1) mean?

Giuseppe Larosa — Fri, 04 Dec 2009 13:37:38 GMT

Hello John,

a good starting point would be to post the named ACL WAN that is denying these packets and the configuration to see if those IP addresses are related to external public ip addresses or ip addresses on multipoint GRE.

also another title for the thread would have been appropriate: I have read too quickly your original post but the title is misleading.

Edit:

>> Where these ICMP packets come from?

they are apparentently sourced by first ip address that appears in the log messages X-> Y, check those devices if they are under your control.

it might be a result that these devices belong to somebody else

Hope to help

Giuseppe

Re: What does ICMP message (11,1) mean?

jgtheodor — Fri, 04 Dec 2009 14:08:02 GMT

You are right Giuseppe,

The title is misleading. Sorry for that honestly.

Now, regarding these IP Addresses: The first one (X) represent the WAN IP Addresses of Spoke Routers and the second one (Y) represent the Loopback IP Address of primary HUB router. These devices are under of my control. It seems that for some reason the spoke routers send these ICMP messages to HUB router. The source IP address is the WAN IP Address Serila Interface and the destination IP Address is the HUB router Loopback IP Address where the VPN IPSec Tunnels are established.

You can see below the Extended ACL that is applied in all WAN interface in all routers in VPN cloud. The network 192.168.192.0 represent the Loopback IP Addresses we use for the IPSec end-points. For example branch A has the Loopback 0 IP Address 192.168.192.10, Branch B 192.168.192.11 etc and finally HUB has 192.168.192.1.

ip access-list extended WAN
permit ahp 192.168.192.0 0.0.15.255 192.168.192.0 0.0.15.255
permit udp 192.168.192.0 0.0.15.255 eq isakmp 192.168.192.0 0.0.15.255 eq isakmp
deny ip any any log

Re: What does ICMP message (11,1) mean?

Giuseppe Larosa — Fri, 04 Dec 2009 14:43:21 GMT

Hello John,

no problems

>> It seems that for some reason the spoke routers send these ICMP messages to HUB router.

I wonder if there are configured IP SLAs on spoke routers that use ICMP probes.

Hope to help

Giuseppe

Re: What does ICMP message (11,1) mean?

jgtheodor — Fri, 04 Dec 2009 15:49:38 GMT

No there are not configured IP SLAs in either HUB & Spoke routers. But as far I know there must be configured in PE routers because our ISP wants to monitor and track the state of WAN links. Do you think that this can cause the relevant ICMP messages? If the answer is yes, why the source IP Address is the WAN IP Address of my router?

Thanks for your help!!!

Re: What does ICMP message (11,1) mean?

marikakis — Sat, 05 Dec 2009 15:24:02 GMT

I am not very familiar with such scenarios, but have a couple of thoughts on this, which might help.
The ICMP type 11 with code 1 means ICMP Time Exceeded/Fragmentation Reassembly Timeout.
If we ignore the possibility of a bug for a moment, to me this means not only that fragmentation is happening,
but also that the routers are doing the reassembly, and they sometimes timeout when doing it. 
Typically the hosts do the reassembly and not the routers, but cases of tunneling are an exception.
When you use IPsec for example, reassembly might be needed at the router so that packet can be decrypted.
Also, the IP addresses in the logs make me think this issue has to do with tunnels between the routers.
Generally the generation of an icmp time-exceeded message might be triggered remotely (e.g. traceroute 
program), but the destination of the icmp time-exceeded typically points to the original device that 
triggered it (of course the traffic traversing the tunnel can still be remote). 
Please take a look at the following document:
http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml#t16
What are the MTU settings of your network? How often do those messages appear? 
Are you monitoring the CPU of your routers and does anything unusual appear there? 

Edit: Sorry for the format of this message, but something goes wrong while editing and haven't managed to fix it up to now.

Re: What does ICMP message (11,1) mean?

jgtheodor — Sun, 06 Dec 2009 18:59:31 GMT

Hello Maria,

The Tunnels which are mGRE (HUB & Spokes) have been configured with MTU 1400 & MSS 1360. I see these log messages many times during the day, mostly in the HUB router. Almost every 5-10 minutes. I have monitored the CPU load not only for the HUB router (3845) but for all the other spoke routers (2811). Nothing unsual or abnormal except those log messages. The IOS Image in 3845 is 12.4(24)T1 and in 2811 routers 12.4(15)T8.

I will check this document carefully and I hope to find something useful. Keep in mind that I have opened a TAC case in Cicso for this behavior but no answer yet since last month.

Thanks Maria!!!

Re: What does ICMP message (11,1) mean?

marikakis — Sun, 06 Dec 2009 20:41:59 GMT

Hi John,

Almost every 5-10 minutes every day (including weekends) or any hours that your network might normally be somewhat silent? Or is there a period that those messages are seen more often? I am asking this because a very regular interval might point to some network management application doing work. Giuseppe already suggested something similar, but I am thinking that an SNMP-based application (SNMP typically uses UDP), might not react to router mss suggestions and might need an extra step to produce equivalent behavior. Other than that, and for the TCP case, I am thinking about your inbound interfaces leading to the tunnels. If you could afford and you think it's worth even though you do not have any other obvious issues besides those messages (and while you are waiting for cisco to respond without excluding the possibility of some bug), maybe you could try to adjust-mss to some inbound interface towards one of the tunnels to see if this makes any difference. I must say however that all those are just theories, and by no means have I managed to conclude on some specific procedure that is causing this.

Kind Regards,

Maria

Re: What does ICMP message (11,1) mean?

jgtheodor — Sun, 06 Dec 2009 21:20:50 GMT

Maria,

The relevant log messages are seen almost every day (including weekends) in HUB router. They are appeared every 5, 10 or 15 minutes every time. Sometimes maybe 30 minutes, I mean it is not a periodic phenomenon, but only in the HUB router. In the spoke routers the relevant log messages are not seen every day. There are spoke routers on which, these log messages have not been appeared for 6 or in some cases for 10 days. In any case the period in which those messages are appeared in spoke routers - when they are appeared - is 1-3 times per day and general not all days.

I read the document you send me. It is great but I just verified that the Tunnel's configuration is correct. I do not use PMTUD, because I have configured static IP MTU and MSS in Tunnel interfaces. I have also already upgraded the HUB router from 12.4(21)T1 to 12.4(24)T1 and I was thinking to proceed in 15.0(1)M.

I am also thinking that it migth useful to follow your suggestion tommorow and configure MSS in LAN interfaces and monitor to see if this makes difference.

In any case I will update the Discussion.

Thanks again and keep walking!

Re: What does ICMP message (11,1) mean?

jgtheodor — Mon, 07 Dec 2009 10:36:47 GMT

Good day again,

After a thorough investigation of HUB router's log messages of the last 6 days, I realized that Maria was right. When the network is somewhat silent during the weekend the number of relevant ICPM packets were too small. I also realized the those messages start appearing every morning at 7:30 - 8:00 and the number of those ICPM packets/ log messages get decreased after 18:00 or 19:00 (branches end day time). It seems that when there is spoke to HUB traffic these ICMP packets are generated for some reason.

Earlier, this morning I added in 2 Branch router's FastEthernet interface the command "ip tcp adjust-mss 1360" and I am monitoring to see if this change makes the difference.

I am also sending you the relevant log messages of the last 6 days in HUB router.

The IP addresses from network 10.195.35.0 are the IP addresses for CE - PE branches WAN connections. The IP Address 192.168.192.1 is the Loopback IP Address in HUB router where the Tunnels from branches are established. The WAN IP Addresses in HUB router is 10.195.35.6 and 10.195.35.10. Do not confuse from two IP Addresses beacuse the Primary HUB router has 2 Serial interfaces. These two Serial interfaces are connected to PE router and the HUB router perform per packet load-balancing as well as the PE router.

Re: What does ICMP message (11,1) mean?

jgtheodor — Mon, 07 Dec 2009 12:55:15 GMT

I would like to update the Discusion with the results from the "ip tcp adjust-mss 1360" command adding in 2 branch router's LAN interfaces:

Nothing at all. I keep taking the same ICMP packets (11,1). I have used the command "debug ip icmp" without any obvious result.

Re: What does ICMP message (11,1) mean?

marikakis — Mon, 07 Dec 2009 15:39:15 GMT

Hi John,

I didn't have a chance to look very closely at the times the ACL messages occur, because you have other messages as well in all the logs you posted, including a severity 1 error. For example:

Dec 05 12:50:04 172.16.250.2 2530095: Dec 5 12:50:03.459: %VPN_HW-1-PACKET_ERROR: slot: 0 Packet Encryption/Decryption error, Output Authentication error:srcadr=192.168.192.33,dstadr=192.168.192.1,size=1504,handle=0x6A1B
Dec 05 12:50:05 172.16.250.2 2530096: Dec 5 12:50:03.459: %CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=4635 local=192.168.192.1 remote=192.168.192.33 spi=FCF866BB seqno=00000061

Please see the following document:

https://supportforums.cisco.com/docs/DOC-2193

Kind Regards,

Maria

Re: What does ICMP message (11,1) mean?

jgtheodor — Mon, 07 Dec 2009 18:39:43 GMT

Hello Maria,

I have opened a TAC case in Cisco for those error log messages two months ago and the TAC Engineer told me that these log messages are just a cosmetic bug and nothing more. Of course the relevant log message does not disrupt the VPN traffic, but If I understand, there are fragment packets in the network which generate the relevant ICMP packets as well as VPN error log messages.

I will try further investigation and update accordingly the Conversation.

Thank you for the great information!

Re: What does ICMP message (11,1) mean?

marikakis — Mon, 07 Dec 2009 19:16:30 GMT

Yes, that's my understanding too. That the ACL log messages and the VPN hardware errors point to fragmentation. That's the only explanation that makes sense to me and puts all the pieces of the discussion together. Reassembly timeouts are caught by ACL and fragmentation is a possibility with those VPN hardware errors. What one considers cosmetic depends. You might not see severe traffic disruption, but there might be actually lost packets in the tunnel when reassembly timeouts occur (maybe because the ACL denies them, maybe anyway). TCP traffic might start with some segments, loses few of them, retransmits and eventually adjusts to the network. This might explain why you see the messages every now and then, especially at higher traffic hours. Also, have in mind that in the first link I posted, it says that fragmentation in the tunnel is possible even if original host sets the DF bit. As the last link I posted suggests, you can either reduce the mss a little bit and/or set manually the DF bit for the tunnel.

Edit: I didn't say clearly that, if any issue exists here, it's not so much a few lost packets, since this is how TCP works anyway, but a potential of a small performance degradation for the routers every now and then depending on the frequency of the messages. And even if this isn't a real issue for your network, we knew that from the beginning based on what you said. Maybe the most serious issue here is to make those messages go away for good so we can all feel better, since nobody likes a router that complains all the time, even if it complains for nothing! Those log messages make the log harder to read and you might miss other issues in the future.

Re: What does ICMP message (11,1) mean?

marikakis — Wed, 09 Dec 2009 12:26:34 GMT

Hi John,

I think you also need to explore if the EIGRP protocol timeouts/goodbyes found in the logs are also associated with this issue. I forgot to ask on a previous post of mine and maybe thought those could be attributed to other unrelated network issues. However, now that I think about it more, I disagree with myself! If the routing protocol timeouts cannot be explained in any other way, then your issue is more serious than a few cosmetic or annoying logs, even if the interruption is usually a couple of seconds (sometimes its more and hold timer takes even more to expire).

Kind Regards,

Maria

Message was edited by: marikakis

Re: What does ICMP message (11,1) mean?

jgtheodor — Wed, 09 Dec 2009 13:14:11 GMT

Yeap,

Maria you are absolutely right. I believe that these EIGRP timeouts are also associated with this issue. I have checked it with our ISP network guys and the routing protocol timeouts cannot be explained at all. They say that there is no any Layer 2 Issue in the MPLS backbone. I have also opened a TAC case in Cisco for this specific issue (3 in total), and remains unanswered. After many EIGRP & Tunnel debugs - no result.

Anyway trying to troubleshoot the issue and after your suggestion, I was wondering if the CEF per packet load-sharing in my HUB router leads to this phenomenon. I am sending you the HUB router's configuration file to take a look. I think that there is no any configuration error.

If you have any suggestion for further troubleshooting, please proceed.

I am still trying to resolve this issue with or without Cisco's help!

Re: What does ICMP message (11,1) mean?

marikakis — Wed, 09 Dec 2009 13:24:53 GMT

Before looking at the config, may I ask if this is a Greek ISP and which one?

Re: What does ICMP message (11,1) mean?

jgtheodor — Wed, 09 Dec 2009 13:40:19 GMT

Yeap this is a Greek ISP, OTE.