10-25-2013 07:15 AM - edited 03-07-2019 04:14 PM
Hello All,
This is my first post so please bare with me as I am unsure of all the proper etiquette. I have a remote site which ties back into my network via a site to site VPN which travels across my ISP's cable modem (a 10Mbps link) and terminates at my ASA. This site has been having issues, especially with their Citrix Desktop connection (uses TCP protocol). At the site I had an 1841 router connected to a 2950 switch. Spanning the trunk port and running Wireshark, I could see that during their Citrix session a packet wouldn't make it to the server and the server would send multiple requests to the client for that missing packet. The Citrix session would hang until the client retransmitted the missing packet. There are only three computers at this site. The users cycle through twice a day for about an hour each time. It is only during those times that errors seem to occur. Looking at the 1841:
Show interface fa0/1
FastEthernet0/1 is up, line protocol is up
Hardware is Gt96k FE, address is 0025.457e.7ba5 (bia 0025.457e.7ba5)
Internet address is 10.xxx.xxx.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:53, output 00:00:00, output hang never
Last clearing of "show interface" counters 20:19:15
Input queue: 0/75/79/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2000 bits/sec, 9 packets/sec
5 minute output rate 53000 bits/sec, 10 packets/sec
350549 packets input, 26089153 bytes
Received 3081 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
294773 packets output, 65495242 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Show buffers:
Buffer elements:
1118 in free list (1119 max allowed)
566595 hits, 0 misses, 619 created
Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 67 @ 1d22h):
50 in free list (20 min, 150 max allowed)
1438573 hits, 502 misses, 24 trims, 24 created
182 failures (0 no memory)
Middle buffers, 600 bytes (total 29, permanent 25, peak 92 @ 2d01h):
27 in free list (10 min, 150 max allowed)
1345094 hits, 822 misses, 137 trims, 141 created
526 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 55 @ 2d02h):
50 in free list (5 min, 150 max allowed)
205653 hits, 105 misses, 5 trims, 5 created
70 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 14 @ 2d01h):
10 in free list (0 min, 100 max allowed)
3342 hits, 283 misses, 23 trims, 23 created
283 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 3 @ 2d01h):
1 in free list (0 min, 10 max allowed)
39 hits, 244 misses, 39 trims, 40 created
244 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 3 @ 2d01h):
1 in free list (0 min, 4 max allowed)
28 hits, 216 misses, 36 trims, 37 created
216 failures (0 no memory)
It seems I am only dropping packets on the link between my switch and my router. The interface to my WAN has no errors. The 2950 also has no errors. What I do not understand is that when I replaced the 2950 with a 3750 (I disabled the L3 with the no ip routing command) for testing purposes, the users no longer experienced their Citrix Desktop sessions freezing. The 1841 continues to report Huge Buffer failures as well as drops on my LAN interface (fa0/1). Why would swapping out the switch solve this problem? I would be extremely appreciative for any information. Also, if you need more information feel free to ask. Thank you!!!
10-28-2013 01:52 AM
Good time of day.
If the switch (on spoke) is the only one and 1841 is the only router, then it's strange to see input queue drops on internal interface.
I would say, your 2950, or even LAN itself had some issue that were causing temporal loops; after you have replaced 2950 with 3750, it could be fixed somehow (by default configuration, or even during recabling).
PS:
It seems I am only dropping packets on the link between my switch and my router
That is not precisely correct; packets could be dropped inside ISP.
PS2: if you are sure your LAN has no issue (but why do they overload 1841?), then you may tune medium buffers.
10-29-2013 06:06 AM
Hi MikhailovskyVV,
So, I went out to the site yesterday because I had a feeling that they were still having problems. Based on all the stats I was getting from the router, they should have still been experiencing their sessions freezing. When I got out there, I found out that they were still having problems. This made me feel a lot better since replacing the switch shouldn't have fixed their issue. The switch is just running a basic vlan 1 config.
PS2: if you are sure your LAN has no issue (but why do they overload 1841?), then you may tune medium buffers.
I did take your advice and tuned the medium buffers yesterday. I also upgraded the router's IOS from 12.4(12b) to 12.4(15)T5. The IOS upgraded didn't seem to make a difference, but tuning the buffers seemed to stabilize the link. Their sessions did not freeze up as often and the router did not register any more drops on the LAN interface or Huge Buffer failures.
I think I increased the medium buffers a little too much though (I have never tuned buffers before), because the server would still occasionally flood the network looking for a packet that it didn't receive from the client. I re-tuned the medium buffers to look like this:
Middle buffers, 600 bytes (total 30, permanent 30, peak 141 @ 17:08:16):
27 in free list (12 min, 175 max allowed)
315218 hits, 852 misses, 211 trims, 211 created
493 failures (0 no memory)
I plan to go back out there this afternoon and deploy a wiretap between my router and the ISP's modem as well as span the port from the switch to the router. I'd like to verify whether or not all the packets are actually leaving the site. I'd love to see if the packet is making it across the network and to the ASA, but it is geographically pretty far away, so this is what I believe to be the best alternative given my location.
Also, this is my first time working in an environment like this, and I am unsure what right looks like. All of my site to site VPN locations running 1841s (with 2 to 5 users on average per site) have drops on the LAN interface as well as approximately 100 huge buffer failures a day. Is this normal?! I know that every remote site I have visited has complained about poor performance and I'd really like to get to the bottom of this and start correcting these issues. Any further advice or guidance would be greatly appreciated! Thank you very much!
10-29-2013 06:50 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
BTW, later router IOSs have a command to autotune buffers, it's the global command "buffers tune automatic".
When working with VPN tunnels, avoiding fragmentation can be helpful. Have you configured your VPN tunnels with Cisco recommendations?
10-29-2013 08:18 AM
Hi JosephDoherty,
Thank you! I had no idea there was an auto tuning feature. Hopefully that will take a lot of the guess work out of how to allocate buffer space. Is it common practice to turn on auto buffer tuning? As for the VPN, we are not fragmenting. Here is a snippet from the 1841 at the site:
crypto pki trustpoint TP-self-signed-23902312
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-23902312
revocation-check none
rsakeypair TP-self-signed-23902312
!
!
crypto pki certificate chain TP-self-signed-23902312
certificate self-signed 01
(Crypto Hash Stuff)
quit
!
crypto isakmp policy 1
encr 3des
authentication pre-share
group 2
crypto isakmp key key2000 address 24.129.xxx.xxx
crypto isakmp keepalive 10
!
crypto ipsec security-association idle-time 86400
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec df-bit clear
!
crypto map SDM_CMAP_1 1 ipsec-isakmp
description Tunnel to 24.129.xxx.xxx
set peer 24.129.xxx.xxx
set security-association lifetime seconds 32400
set transform-set ESP-3DES-SHA
match address 110
!
!
!
ip tcp synwait-time 10
buffers tune automatic
!
!
!
interface FastEthernet0/0
description $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-FE 0$$ES_LAN$$FW_INSIDE$
ip address 71.40.xxx.xxx 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip tcp adjust-mss 1452
duplex auto
speed auto
no mop enabled
crypto map SDM_CMAP_1
crypto ipsec df-bit clear
!
interface FastEthernet0/1
ip address 10.216.xxx.xxx 255.255.255.0
ip helper-address 10.110.xxx.xxx
ip helper-address 10.210.xxx.xxx
no ip redirects
no ip unreachables
no ip proxy-arp
speed 100
full-duplex
no mop enabled
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 71.40.xxx.xxx
I am not sure what Cisco recommends for VPN tunnels but I will look it up to see if we are setting them up right. I am fairly new to VPN tunnels. Thank you for your help!
10-29-2013 08:28 AM
Hello.
I guess concern about fragmentation could be correct. And this definitely must be fixed.
You need to add "ip tcp adj 1360" on your internal interface.
Anyway, the drops on input queue of internal interface has nothing common with IPSec fragmentation... at least for a network of 3 clients. I guess LAN is not healthy. Try to apply "storm-control broadcast 3.0" on every interface of the switch, add port-security limiting maximum number of MACs learnt to 4; add BPDUguard feature to all ports.
I would suggest to analyse what traffic is flowing to 1841 during the issue. I would also guess that WireShark is not a best tool to capture the traffic on SPAN port (I had a chance to span port of "Orbit Downloader 4.x" issue when it was flooding a lot of TCP SYN - up to 70K packets per second... Wireshark could catch no more than 10 per second!)
PS: why do you have "no ip unreachable" on internal interface?
PS2: why don't you have ACL applied on public interface?
10-29-2013 09:15 AM
Hello MikhailovskyVV,
I really do appreciate your time and help on this issue. I will implement the commands on the switch. I want to rule out anything that can pose a problem. As for the Wireshark, do you have another method of packet capture that you would suggest? Would using tshark (I believe that's what the command line is called) to capture the packets then doing a dump afterwards be a better approach?
Included is a snippet of the last Wireshark capture I took yesterday. 10.210.xxx.xxx is the Citrix server and 10.216.xxx.xxx is the client.
I am not sure if that helps any. This capture was after I made adjustments to the middle buffers.
10-29-2013 11:51 AM
Sorry, the pictures miss a lot of information.
If it's possible, please paste capture file (please, ommit begining of the session).
Also background traffic could help to troubleshoot... try to execute ping in the background (between client and server), so we could see icmp traffic.
PS: no idea what to use instead of Wireshark... maybe Microsoft NetMon, but I had no chance to check it with Orbit Downloader.
10-29-2013 01:16 PM
Hello, Jason.
During the issue, could you please run following commands on the router:
sh ip int f0/0
sh ip int f0/1
sh proc cpu hist
sh proc cpu sort 1
sh crypto isakmp sa
sh ip cef
sh mem
sh ver (just for the whole picture)
PS: I guess there could be one more point - the router might be missing ip cef command.
10-29-2013 05:27 PM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
I don't know how common using auto buffer tunning is. My guess would be few are aware of this feature.
Your mss-adjust value, I believe, is too large. Here's a Cisco white paper addressing MTU configurations for tunnels: http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide