cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
984
Views
0
Helpful
9
Replies

Help; performance increase when swapped 2950 with 3750?

pagan1104
Level 1
Level 1

Hello All,

This is my first post so please bare with me as I am unsure of all the proper etiquette. I have a remote site which ties back into my network via a site to site VPN which travels across my ISP's cable modem (a 10Mbps link) and terminates at my ASA. This site has been having issues, especially with their Citrix Desktop connection (uses TCP protocol). At the site I had an 1841 router connected to a 2950 switch. Spanning the trunk port and running Wireshark, I could see that during their Citrix session a packet wouldn't make it to the server and the server would send multiple requests to the client for that missing packet. The Citrix session would hang until the client retransmitted the missing packet. There are only three computers at this site. The users cycle through twice a day for about an hour each time. It is only during those times that errors seem to occur. Looking at the 1841:

Show interface fa0/1

FastEthernet0/1 is up, line protocol is up

  Hardware is Gt96k FE, address is 0025.457e.7ba5 (bia 0025.457e.7ba5)

  Internet address is 10.xxx.xxx.1/24

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, 100BaseTX/FX

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:53, output 00:00:00, output hang never

  Last clearing of "show interface" counters 20:19:15

  Input queue: 0/75/79/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 2000 bits/sec, 9 packets/sec

  5 minute output rate 53000 bits/sec, 10 packets/sec

     350549 packets input, 26089153 bytes

     Received 3081 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog

     0 input packets with dribble condition detected

     294773 packets output, 65495242 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier

     0 output buffer failures, 0 output buffers swapped out

Show buffers:

Buffer elements:

     1118 in free list (1119 max allowed)

     566595 hits, 0 misses, 619 created

Public buffer pools:

Small buffers, 104 bytes (total 50, permanent 50, peak 67 @ 1d22h):

     50 in free list (20 min, 150 max allowed)

     1438573 hits, 502 misses, 24 trims, 24 created

     182 failures (0 no memory)

Middle buffers, 600 bytes (total 29, permanent 25, peak 92 @ 2d01h):

     27 in free list (10 min, 150 max allowed)

     1345094 hits, 822 misses, 137 trims, 141 created

     526 failures (0 no memory)

Big buffers, 1536 bytes (total 50, permanent 50, peak 55 @ 2d02h):

     50 in free list (5 min, 150 max allowed)

     205653 hits, 105 misses, 5 trims, 5 created

     70 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 14 @ 2d01h):

     10 in free list (0 min, 100 max allowed)

     3342 hits, 283 misses, 23 trims, 23 created

     283 failures (0 no memory)

Large buffers, 5024 bytes (total 1, permanent 0, peak 3 @ 2d01h):

     1 in free list (0 min, 10 max allowed)

     39 hits, 244 misses, 39 trims, 40 created

     244 failures (0 no memory)

Huge buffers, 18024 bytes (total 1, permanent 0, peak 3 @ 2d01h):

     1 in free list (0 min, 4 max allowed)

     28 hits, 216 misses, 36 trims, 37 created

     216 failures (0 no memory)

It seems I am only dropping packets on the link between my switch and my router. The interface to my WAN has no errors. The 2950 also has no errors. What I do not understand is that when I replaced the 2950 with a 3750 (I disabled the L3 with the no ip routing command) for testing purposes, the users no longer experienced their Citrix Desktop sessions freezing. The 1841 continues to report Huge Buffer failures as well as drops on my LAN interface (fa0/1). Why would swapping out the switch solve this problem? I would be extremely appreciative for any information. Also, if you need more information feel free to ask. Thank you!!!

9 Replies 9

Good time of day.

If the switch (on spoke) is the only one and 1841 is the only router, then it's strange to see input queue drops on internal interface.

I would say, your 2950, or even LAN itself had some issue that were causing temporal loops; after you have replaced 2950 with 3750, it could be fixed somehow (by default configuration, or even during recabling).

PS:

It seems I am only dropping packets on the link between my switch and my router

That is not precisely correct; packets could be dropped inside ISP.

PS2: if you are sure your LAN has no issue (but why do they overload 1841?), then you may tune medium buffers.

Hi MikhailovskyVV,

So, I went out to the site yesterday because I had a feeling that they were still having problems. Based on all the stats I was getting from the router, they should have still been experiencing their sessions freezing. When I got out there, I found out that they were still having problems. This made me feel a lot better since replacing the switch shouldn't have fixed their issue. The switch is just running a basic vlan 1 config.

PS2: if you are sure your LAN has no issue (but why do they overload 1841?), then you may tune medium buffers.

I did take your advice and tuned the medium buffers yesterday. I also upgraded the router's IOS from 12.4(12b) to 12.4(15)T5. The IOS upgraded didn't seem to make a difference, but tuning the buffers seemed to stabilize the link. Their sessions did not freeze up as often and the router did not register any more drops on the LAN interface or Huge Buffer failures.

I think I increased the medium buffers a little too much though (I have never tuned buffers before), because the server would still occasionally flood the network looking for a packet that it didn't receive from the client. I re-tuned the medium buffers to look like this:

Middle buffers, 600 bytes (total 30, permanent 30, peak 141 @ 17:08:16):

     27 in free list (12 min, 175 max allowed)

     315218 hits, 852 misses, 211 trims, 211 created

     493 failures (0 no memory)

I plan to go back out there this afternoon and deploy a wiretap between my router and the ISP's modem as well as span the port from the switch to the router. I'd like to verify whether or not all the packets are actually leaving the site. I'd love to see if the packet is making it across the network and to the ASA, but it is geographically pretty far away, so this is what I believe to be the best alternative given my location.

Also, this is my first time working in an environment like this, and I am unsure what right looks like. All of my site to site VPN locations running 1841s (with 2 to 5 users on average per site) have drops on the LAN interface as well as approximately 100 huge buffer failures a day. Is this normal?! I know that every remote site I have visited has complained about poor performance and I'd really like to get to the bottom of this and start correcting these issues. Any further advice or guidance would be greatly appreciated! Thank you very much!

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

BTW, later router IOSs have a command to autotune buffers, it's the global command "buffers tune automatic".

When working with VPN tunnels, avoiding fragmentation can be helpful.  Have you configured your VPN tunnels with Cisco recommendations?

Hi JosephDoherty,

Thank you! I had no idea there was an auto tuning feature. Hopefully that will take a lot of the guess work out of how to allocate buffer space. Is it common practice to turn on auto buffer tuning? As for the VPN, we are not fragmenting. Here is a snippet from the 1841 at the site:


crypto pki trustpoint TP-self-signed-23902312

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-23902312

revocation-check none

rsakeypair TP-self-signed-23902312

!

!

crypto pki certificate chain TP-self-signed-23902312

certificate self-signed 01

        (Crypto Hash Stuff)

        quit

!

crypto isakmp policy 1

encr 3des

authentication pre-share

group 2

crypto isakmp key key2000 address 24.129.xxx.xxx

crypto isakmp keepalive 10

!

crypto ipsec security-association idle-time 86400

!

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec df-bit clear

!

crypto map SDM_CMAP_1 1 ipsec-isakmp

description Tunnel to 24.129.xxx.xxx

set peer 24.129.xxx.xxx

set security-association lifetime seconds 32400

set transform-set ESP-3DES-SHA

match address 110

!

!

!

ip tcp synwait-time 10

buffers tune automatic

!

!

!

interface FastEthernet0/0

description $ETH-LAN$$ETH-SW-LAUNCH$$INTF-INFO-FE 0$$ES_LAN$$FW_INSIDE$

ip address 71.40.xxx.xxx 255.255.255.248

no ip redirects

no ip unreachables

no ip proxy-arp

ip tcp adjust-mss 1452

duplex auto

speed auto

no mop enabled

crypto map SDM_CMAP_1

crypto ipsec df-bit clear

!

interface FastEthernet0/1

ip address 10.216.xxx.xxx 255.255.255.0

ip helper-address 10.110.xxx.xxx

ip helper-address 10.210.xxx.xxx

no ip redirects

no ip unreachables

no ip proxy-arp

speed 100

full-duplex

no mop enabled

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 71.40.xxx.xxx

I am not sure what Cisco recommends for VPN tunnels but I will look it up to see if we are setting them up right. I am fairly new to VPN tunnels. Thank you for your help!

Hello.

I guess concern about fragmentation could be correct. And this definitely must be fixed.

You need to add "ip tcp adj 1360" on your internal interface.

Anyway, the drops on input queue of internal interface has nothing common with IPSec fragmentation... at least for a network of 3 clients. I guess LAN is not healthy. Try to apply "storm-control broadcast 3.0" on every interface of the switch, add port-security limiting maximum number of MACs learnt to 4; add BPDUguard feature to all ports.

I would suggest to analyse what traffic is flowing to 1841 during the issue. I would also guess that WireShark is not a best tool to capture the traffic on SPAN port (I had a chance to span port of "Orbit Downloader 4.x" issue when it was flooding a lot of TCP SYN - up to 70K packets per second... Wireshark could catch no more than 10 per second!)

PS: why do you have "no ip unreachable" on internal interface?

PS2: why don't you have ACL applied on public interface?

Hello MikhailovskyVV,

I really do appreciate your time and help on this issue. I will implement the commands on the switch. I want to rule out anything that can pose a problem. As for the Wireshark, do you have another method of packet capture that you would suggest? Would using tshark (I believe that's what the command line is called) to capture the packets then doing a dump afterwards be a better approach?

Included is a snippet of the last Wireshark capture I took yesterday. 10.210.xxx.xxx is the Citrix server and 10.216.xxx.xxx is the client.

I am not sure if that helps any. This capture was after I made adjustments to the middle buffers.

Sorry, the pictures miss a lot of information.

If it's possible, please paste capture file (please, ommit begining of the session).

Also background traffic could help to troubleshoot... try to execute ping in the background (between client and server), so we could see icmp traffic.

PS: no idea what to use instead of Wireshark... maybe Microsoft NetMon, but I had no chance to check it with Orbit Downloader.

Hello, Jason.

During the issue, could you please run following commands on the router:

sh ip int f0/0

sh ip int f0/1

sh proc cpu hist

sh proc cpu sort 1

sh crypto isakmp sa

sh ip cef

sh mem

sh ver (just for the whole picture)

PS: I guess there could be one more point - the router might be missing ip cef command.

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

I don't know how common using auto buffer tunning is.  My guess would be few are aware of this feature.

Your mss-adjust value, I believe, is too large.  Here's a Cisco white paper addressing MTU configurations for tunnels: http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking products for a $25 gift card