ping Quench phenomenon

ricewu2006 · ‎08-13-2015

Hi, guys,

I ran into a weird situation here:

From the gateway switch with interface vlan 500 and 501, I pinged one destination in vlan501 and vlan500 respectively. The results are as follows:

====================

switch#ping 3.27.113.230 repeat 500 size 1024

Type escape sequence to abort.
Sending 500, 1024-byte ICMP Echos to 3.27.113.230, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQ
Success rate is 29 percent (148/500), round-trip min/avg/max = 1/1/4 ms

======================

I've read some documents which says that this Q results mean the server buffer is full, telling the source switch not to send packets any more.

The server team has checked the server, claiming that their server has no problems. It has a lot of applications running on that too.

I've checked there are quite a few hosts on the vlan 500 and 501 through show ip arp. Ping them with size 1024 was all normal.

What should be the next step? Maybe the cable between the switch and the host has problems? Or maybe the NICs on the server need to be checked again?

Thanks!

Peter Paluch · ‎08-13-2015

Hi,

The ICMP Source Quench message was originally intended to allow routers or end hosts inform sources of packets that they should slow down. See RFC 792 Page 10 for more detail. However, the use and implementation of Source Quench has never been widespread, and there were significant doubts about its usability and even security. Therefore, in RFC 1812 Section 4.3.3.3, sending of Source Quench messages by routers has been deprecated, and furthermore, RFC 6633 deprecates the sending of Source Quench messages altogether, both by routers and end hosts. As a side note, RFC 5927 named ICMP Attacks against TCP nicely describes a number of Source Quench-based attacks against a TCP session.

All of this is to show one thing: ICMP Source Quench has been considered deprecated since 1995 and its use has been rare even before then, not to mention today. I am extremely surprised you actually have a device in your network that is capable of sending these Source Quench messages.

In any case, I do not believe there is a case for concern. Sending ICMP Source Quench messages is up to the system that tries to tell the source to slow down, up to its own logic and decision. A Source Quench does not necessarily mean that router or recipient is overloaded. It may simply mean that it wishes the source to send packets more slowly. It is quite possible that the device you have been pinging didn't like the relatively high rate of incoming pings, and started sending Source Quench messages to tell you to slow down.

Cisco routers running IOSes later than 12.0 do not support sending Source Quench messages:

http://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/28745-44.html#qa5

Therefore, while a Source Quench can be sent by any device along the path to the destination, I suppose that this Quench message was originated by the end host. You should do some packet capturing, though, to verify whether this is really the case - seeing ANY device sending a Source Quench nowadays is a small miracle so I would be VERY interested what operating system still sends those.

So my suggestion is - don't worry about this too much, just make sure that the sender IP address of these Source Quench messages is truly the address of the device you were pinging. If it is, just ignore it - it may very well be some kind of throttling mechanism for incoming pings.

Best regards,
Peter

ricewu2006 · ‎08-13-2015

wa! Thanks, Peter. So detailed!

The reason why I paid attention to this phenomenon is the connection slowness complained by the user.

3.27.113.230 is actually the source connected to the switch.

3.151.20.13 is the destination server connected to another switch.

From 27.113.230 to 151.20.13, the traffic is quite slow. And the ping results show many Q's. So we thought there might be something wrong with the network.

I did some troubleshooting.

1. Just like what I wrote, I ping 3.27.113.230 from the gateway switch, many Q's.

2. Then, I add something here. I ping 3.151.20.13 from its gateway switch. Still many Q's.

3. However, I ping the gateway of 3.151.20.13 (say, 3.151.20.1) from the gateway of 3.27.113.230 (say, 3.27.113.1) on the switch. Almost no packet lost! From this step3, I think the backbone WAN link between the host and the server should be fine.

The issue might still be on the servers.

Any ideas? :)

Peter Paluch · ‎08-13-2015

Hi Rice,

Unfortunately, ping is only a tool to test basic connectivity but it is not suitable for testing the load or the capacity of a link or a server. Furthermore, you should not consider the results from a ping as being representative of the entire system performance. Different operating systems may choose to artificially slow down responses to pings to make sure they don't waste much time on handling unnecessary traffic.

In both your cases, you have pinged a server from a Cisco device. Cisco pings are being sent out relatively quickly, and the servers may be intentionally configured to avoid responding to pings so fast. That is why you may be getting the Quench messages. The fact that you pinged between the two switches with almost no packet loss is somewhat of an assurance that it is not the WAN link that causes trouble. Cisco devices are, in general, happy to respond to pings quickly.

With pings being generally an unreliable indicator of a performance issue, we do not have much to diagnose a potential performance problem. Those Source Quench messages may or may not be related. I suggest doing this:

Configure a SPAN session on the switch if possible, and while doing the ping that causes Source Quench messages to come back, capture this stream of Source Quench messages using Wireshark. What is extremely important to find out is who is the sender of these Source Quench messages. Be careful when analyzing the stream: An ICMP message is carried in an IP packet, but in the ICMP message body, there is the header of the packet that was undeliverable and to which this ICMP message is related. We need to know who was the sender of the ICMP message itself (that is, the outer IP header source address). Do not confuse the address in the outside header with the address inside the ICMP body - we're not interested in the address inside the ICMP body as we already know who that is - it's the device from which you're pinging. We need the source address from the outer IP header in which the ICMP Source Quench is carried.
If there is an option of running Wireshark on the computer of the user that has complained about the slowness, try doing that and see if the user's computer also receives Source Quench messages while doing normal communication with the server. Do not perform pings from the user's computer, just let him do the normal work. If that normal work elicits Source Quench messages then the configuration of their source has to be very closely inspected.

Best regards,
Peter

ricewu2006 · ‎08-13-2015

Many thanks!

I've made a detailed summary to the user and other teams.After I got the results, I'll keep you informed.