Huge IPSEC+GRE overhead - average packet size?

[ValentinPuiu91087]

Hello everyone,

During some time intervals when Webex sessions are held, I have a huge difference between the tunnel and physical interface reported traffic. Note that all my traffic is being tunneled over this interface.

While the physical interface reports a bandwidth of ~50 Mbps, the tunnel interface reports an utilization of ~30 Mbps. I assume the difference is packet decapsulation (incoming traffic) of small packets in which there's big overhead.

Is there a way I can find the average packet size? Can this be proven?

I have an old router platform, CISCO2951/K9 version 15.6(3)M9.

Thank you!
Vali Puiu

[Joseph W. Doherty]

The difference in bandwidth between a tunnel interface and a physical interface, possibly being because of the VPN overhead, sounds likely.  Further, if Webex is using UDP, max size packets would be fragmented, also increasing the VPN overhead.  (If you're using something like MSS adjust, this would avoid fragmentation for TCP traffic, not UDP taffic.)

As to "seeing" packets sizes, I recall (years ago) seeing stats where there were counts for different sized frames crossing an interface.  Don't recall, though, what command or on what platform I saw these.  Might have been Netflow, might have been RMON type stats, or traffic stats using another traffic monitoring interface command (I recall there having been such a command, but not the command itself).

You might also see about using some form of packet capture, which, possibly, your IOS supports.

[Georg Pauwen]

Hello,

I remember something about having to set the bandwidth on the tunnel using the 'bandwidth' command, and that bandwidth needs to be the speed of the physical interface, otherwise the counters on the physical and the tunnel interfaces won't match.

So what if you set the 'bandwidth' on the tunnel to whatever the physical interface is using (e.g. 100Mbps) ?

[Joseph W. Doherty]

NB: BTW, I don't recall bumping into that.  If you did, perhaps a bug?

[paul driver]

Hello

Your Ipsec/gre overhead total would vary depending on what other additonal overheads youve added to the orignal packet size such as authentication/encryption types, NAT-T etc..which could increase to a total packet side of upto 1630 and if using ipv6 an additional 40 bytes on top.
You could  try to perfrom a ping sweep to find you actual framentation ceiling..

Example:

ping
Protocol [ip]:
Target IP address: y.y.y.y
Repeat count [5]: 2 
Datagram size [100]:1500
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: x,x,x,x
Type of service [0]:
Set DF bit in IP header? [no]: y
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]: y
Sweep min size [36]: 1450 <------------------------------------start of range
Sweep max size [18024]: 1635<----------------------------------end of range
Sweep interval [1]: 10 <----------------------------------------increment of each ping

[valentin.puiu92]

The bandwidth is configured as 50 on both the physical and tunnel interfaces - to have a proper reporting from our tools.

What will confirm this theory is to know the average packet size. I will continue the search.

Will let you know if something was discovered.

Thank you for your help!

[valentin.puiu92]

And I forgot to add something. My memory utilization (not CPU) is in average 93%. Can this impact the encapsulation/decapsulation process?

bits/sec	34374000		bits/packets	8493.699
packets/sec	4047
			bytes/packet	1061.712

Took the live utilization but the packet size is not what I expected to be.

[Giuseppe Larosa]

Hello @valentin.puiu92 ,

are your Webex conference calls using also Video or only audio ?

if using Video you will have the audio stream that is actually made of small packets and for them the overhead is high.

Then there are Video stream packets and these are big packets for them the IPSec + GRE has less effect for overhead but could cause fragmentation.

Then there also control packets of the Webex session ( the control protocols are used  to understand who is speaking in this moment and to have the correct webcam to show his/her face and also in multi parties calls there can be the need to put in foreground the active speaker and so on)

There are also text based chats with the possibility to attach file to send to someone and this again lead to big packets for the file transfer. ( these should be TCP based as are one to one).

>> My memory utilization (not CPU) is in average 93%. Can this impact the encapsulation/decapsulation process?

If you are not experiencing sudden delays or losses it is not impacting for the moment .

Of course you need to take care of memory usage over time.

if it increases every day or over time you have likely a SW bug that causes memory leakage and that in the long term will need a reload to fix it for some time.

Hope to help

Giuseppe