cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1797
Views
0
Helpful
11
Replies

2651XM: Slower Network Routing only with OS X

Ian Stephens
Level 1
Level 1

So we have a little problem that we thought we'd post up on these forums to try to get a result and hopefully a solution.

We have a Cisco 2651XM at the edge of our network, which routes our public IP block. This sits on a 100 Mbit/s ethernet pipe (full duplex) up in the datacenter. We are also running 100mb full duplex on our our side of the network. We have several public servers behind the router.

We have recently set-up a new Apple OS X Lion Server to serve a few websites. However, when downloading some files from the server the from a remote location, I noticed I can only get a maximum of 15 Mbit/s out of the connection.

When downloading a large file over HTTP (Apache) from this OS X Lion server from another server on the same internal network (behind the router), we get a full 100 Mbit/s transfer speed.

However, when we download the same file from anywhere on the internet (external side of the router), we can only manage to get 15 Mbit/s out of the transfer.

However, we also have other Linux & Windows servers that we can achieve a full 50 Mbit/s (our office connection speed) externally under the same network conditions.

So it also appears it's only for a single connection - not a limitation for the whole server. If I open two HTTP connections, I can get say 15 Mbit/s out of each transfer - totalling 30 Mbit/s... if that makes sense.

Any ideas on this strange issue? Your thoughts, suggestions and tips are most appreciated.

Update:

I also notice slower ping latency when pinging from outside the network. Most of the servers reply in 15ms while the new OS X server usually takes over 40ms on average.

Router show ver:

Cisco Internetwork Operating System Software

IOS (tm) C2600 Software (C2600-IS-M), Version 12.2(11)YT2, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

TAC Support: http://www.cisco.com/tac

Copyright (c) 1986-2003 by cisco Systems, Inc.

Compiled Thu 27-Feb-03 17:33 by cmong

Image text-base: 0x80008098, data-base: 0x818BA404

ROM: System Bootstrap, Version 12.2(8r) , RELEASE SOFTWARE (fc1)

ROM: C2600 Software (C2600-IS-M), Version 12.2(11)YT2, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

r1.blahblah.com uptime is 29 weeks, 4 days, 21 hours, 44 minutes

System returned to ROM by power-on

System restarted at 22:15:04 UTC Tue Jan 10 2012

System image file is "flash:c2600-is-mz.122-11.YT2.bin"

cisco 2651XM (MPC860P) processor (revision 0x400) with 60416K/5120K bytes of memory.

Processor board ID JHY0852K145 (1881713462)

M860 processor: part number 5, mask 2

Bridging software.

X.25 software, Version 3.0.0.

2 FastEthernet/IEEE 802.3 interface(s)

32K bytes of non-volatile configuration memory.

16384K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

1 Accepted Solution

Accepted Solutions

Peter Paluch
Cisco Employee
Cisco Employee

Hello Ryan,

This is an interesting issue indeed! The fact that different OSes behave differently points to a possible implementation difference in the TCP behavior or default settings, and I guess this is the place we should start investigating.

My immediate thought relates to the MTU and TCP MSS setting of the OS X. To what values are they set currently? Ideally, the MTU should be 1500B, the TCP MSS should be 1460B. Are you able to verify that? As I discussed this issue with Jan Hrnko, he has pointed me to an article saying:

Mac OS X has a very small default TCP packet size, 512 bytes. You are recommended to increase it to the Ethernet default of 1500 bytes, which corresponds to a maximum segment size of 1460 bytes.

http://www.macfreek.nl/memory/Kernel_Configuration#Packet_Size

Besides that, there are lots of documents to be found on Google related to "mac os x tcp tuning". I believe they are also worth reading. I understand that the OS X behaves differently when talking to a local client and when talking to a client behind the router, but in more advanced OSes, the MTU/MSS can sometimes be set on a per-destination basis, or at least they may differ for local and off-subnet hosts.

I also suggest running Wireshark on the OS X server and recording the complete transcript of a TCP session that exhibits the 15Mbps bottlenecks. Afterwards, look for any unusual data like TCP retransmissions, duplicates, duplicate ACKs, etc. Especially pay attention to the MSS negotiated by both parties in the TCP connection (check the first three segments here for the MSS value) and also check the TCP window size behavior as indicated in TCP segments. Also check if ECN is being negotiated and whether the TCP are encapsulated in IP packets with DF bit set (possible issues with Path MTU Discovery here). Lots of thoughts... perhaps it would indeed be best to record the impaired TCP session using Wireshark and post it somewhere so that we can have a look.

Also, do you believe you could post your 2651XM router configuration (with sensitive information removed)?

Thank you!

Best regards,

Peter

View solution in original post

11 Replies 11

Peter Paluch
Cisco Employee
Cisco Employee

Hello Ryan,

This is an interesting issue indeed! The fact that different OSes behave differently points to a possible implementation difference in the TCP behavior or default settings, and I guess this is the place we should start investigating.

My immediate thought relates to the MTU and TCP MSS setting of the OS X. To what values are they set currently? Ideally, the MTU should be 1500B, the TCP MSS should be 1460B. Are you able to verify that? As I discussed this issue with Jan Hrnko, he has pointed me to an article saying:

Mac OS X has a very small default TCP packet size, 512 bytes. You are recommended to increase it to the Ethernet default of 1500 bytes, which corresponds to a maximum segment size of 1460 bytes.

http://www.macfreek.nl/memory/Kernel_Configuration#Packet_Size

Besides that, there are lots of documents to be found on Google related to "mac os x tcp tuning". I believe they are also worth reading. I understand that the OS X behaves differently when talking to a local client and when talking to a client behind the router, but in more advanced OSes, the MTU/MSS can sometimes be set on a per-destination basis, or at least they may differ for local and off-subnet hosts.

I also suggest running Wireshark on the OS X server and recording the complete transcript of a TCP session that exhibits the 15Mbps bottlenecks. Afterwards, look for any unusual data like TCP retransmissions, duplicates, duplicate ACKs, etc. Especially pay attention to the MSS negotiated by both parties in the TCP connection (check the first three segments here for the MSS value) and also check the TCP window size behavior as indicated in TCP segments. Also check if ECN is being negotiated and whether the TCP are encapsulated in IP packets with DF bit set (possible issues with Path MTU Discovery here). Lots of thoughts... perhaps it would indeed be best to record the impaired TCP session using Wireshark and post it somewhere so that we can have a look.

Also, do you believe you could post your 2651XM router configuration (with sensitive information removed)?

Thank you!

Best regards,

Peter

Thank you for your reply Peter.

I have looked into the TCP packet size on the OS X machine, and can confirm it is set at 512 bytes:

web7:~ ryanbarclay$ sysctl -a | grep mssdflt

net.inet.tcp.mssdflt: 512

net.inet.tcp.v6mssdflt: 1024

However, the MTU is correctly set at 1500 bytes.  I will up the packet size to 1460 and test the results.  I'll keep this post updated!

Here is the router config - it's a very simple config, nothing fancy; no traffic shaping... etc:

!

! Last configuration change at 20:44:33 UTC Thu Apr 5 2012

! NVRAM config last updated at 20:44:36 UTC Thu Apr 5 2012

!

version 12.2

service timestamps debug uptime

service timestamps log uptime

service password-encryption

!

hostname r1.blahblah.com

!

no logging console

enable password 7 xxxxxxxx

!

ip subnet-zero

!

!

ip name-server xxx.xxx.xxx.xxx

!       

!        

mta receive maximum-recipients 0

!        

!               

!        

interface FastEthernet0/0

description connected to RBFTP Networks

ip address xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx

duplex auto

speed auto

!        

interface FastEthernet0/1

description connected to GXN Connection

ip address xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx

duplex auto

speed auto

!        

ip classless

ip route 0.0.0.0 0.0.0.0 FastEthernet0/1

no ip http server

!        

!        

access-list 12 permit xxx.xxx.xxx.xxx

access-list 12 permit xxx.xxx.xxx.xxx

access-list 12 permit xxx.xxx.xxx.xxx

access-list 13 permit xxx.xxx.xxx.xxx

access-list 13 permit xxx.xxx.xxx.xxx

access-list 13 permit xxx.xxx.xxx.xxx

!        

snmp-server community public RO 13

snmp-server location London, Docklands, United Kingdom

snmp-server contact blah blah blah

snmp-server enable traps tty

call rsvp-sync

!        

!        

mgcp profile default

!        

!        

!        

dial-peer cor custom

!        

!        

!        

banner motd ^CBlah Site Router

London Docklands, UK^C

!        

line con 0

exec-timeout 0 0

password 7 xxxxxxx

login   

line aux 0

line vty 0 4

access-class 12 in

password 7 xxxxxxx

login   

!        

ntp clock-period 17208074

ntp server xxx.xxx.xxx.xxx

!        

end

Okay Peter, we have some progress!

After upping the MSS size to 1460 and rebooting, I was able to get over 30 Mbit/s out of a single HTTP connection - instead of the dreaded 15 Mbit/s!

This is great, and the speed is on par with our other servers now.  It seemed like that was the problem.

However, if I open up several HTTP connections for the same file being downloaded, I can max out our office bandwidth at 50 Mbit/s.  Why is a single connection still limited to 30 Mbit/s?  Is there any additional tuning I can implement to allow a single connection to achieve "full" speed?

These are our current window size settings on the OS X box:

net.inet.tcp.sendspace: 65536

net.inet.tcp.recvspace: 65536

net.inet.udp.recvspace: 42080

net.inet.raw.recvspace: 8192

Thank you for your help, it's most appreciated.

Update:

It appears that OS X versions above 10.5 auto-tune the window sizes.  So adjusting the defaults wouln't actually change anything.

Any ideas guys?

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Peter has already mentioned, but want to emphasis checking what TCP receive window size is being advertized.

Ian Stephens
Level 1
Level 1

Perhaps I should have actually done some testing before posting.

I can now achieve over 50 Mbit/s on a single connection with OS X.

This problem is now fixed thanks to the following settings in /etc/sysctl.conf

net.inet.tcp.mssdflt=1460

net.inet.tcp.sendspace=262144

net.inet.tcp.recvspace=262144

Thanks for your help guys, it lead me in the right direction to fixing this.

Hello Ryan,

Thanks for letting us know what settings you modified to get a higher output out of your OS X server!

One comment to your router's configuration, though: your default route is currently defined as

ip route 0.0.0.0 0.0.0.0 FastEthernet0/1

This is strongly discouraged - without going into much details, this will lead to huge ARP tables and excessive ARP traffic on the Fa0/1 interface. We have even heard of routers restarting themselves because of memory exhaustion thanks to ARP table bloat. Correctly, if configuring static routing entries over multiaccess interfaces such as Ethernet, you should always use next-hop address, i.e.

ip route 0.0.0.0 0.0.0.0 A.B.C.D

where A.B.C.D is the next-hop IP address. I strongly encourage you to correct your configuration in this aspect. Be careful when removing the current default route - connectivity to the internet will be internet until the new default route is added correctly.

Best regards,

Peter

Thanks for making me aware of this Peter. I'll make the suggested change to the config. I think I'll do it while up at the datacenter from inside the network so that I don't get kicked out if it doesn't work!

While I'm doing that, are there any other config directives that you can recommend for increased performance?

I've never been on any Cisco courses or any networking courses for that matter; my knowledge is entirely from research so it's interesting learning about these things!

Sent from Cisco Technical Support iPad App

Hello Ryan,

I apologize for answering late.

I do not see any other issue in your configuration requiring correction. Just one hint - please verify if Cisco Express Forwarding (an improved organization of the routing table and adjacent data structures optimized for fast lookups) is activated using the show ip cef command. If this command produces a listing of networks similar to your routing table then CEF is working. If it says that CEF is not running, you should try to activate it using the ip cef global configuration level command.

Best regards,

Peter

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Ryan Barclay wrote:

Perhaps I should have actually done some testing before posting.

I can now achieve over 50 Mbit/s on a single connection with OS X.

This problem is now fixed thanks to the following settings in /etc/sysctl.conf

net.inet.tcp.mssdflt=1460

net.inet.tcp.sendspace=262144

net.inet.tcp.recvspace=262144

Thanks for your help guys, it lead me in the right direction to fixing this.

BTW, there's a chance you don't need to adjust tcp.sendspace.

Thanks for the response Joseph. Do you think it's worth doing that then; would there be any benefit?

Sent from Cisco Technical Support iPad App

Disclaimer


The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

Yes there might be benefit, but my understanding, hosts can usually draw from sending app quick enough they don't usually need to allocate a large send buffer.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: