Cisco 4500x pair throughput issue on 10G connection

[Chris Rug]

Hello all,

i have a pair of 4500X, connected with 10G LR SFP+

They are connected via an ACCESS VLAN.

On both side i have two pretty new (and daily used) notebooks used to measure throughput.

While measuring i always observe a disbalance from side A to side B resulting in throughput 400-600 Meg compared to 1000 Meg on the other side so:

NB A measures 400-600 Downstream and 1000 Upstream

NB B measures 1000 Downstream and 400-600 Upstream.

The switches share this software version: 

A

15.0(1r)SG11 03.11.02.E

B

15.0(1r)SG12 03.11.03a.E

Buffers

Public buffer pools:
Small buffers, 104 bytes (total 1000, permanent 1000):
1000 in free list (20 min, 150 max allowed)
386428387 hits, 25 misses, 75 trims, 75 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 500, permanent 500):
500 in free list (10 min, 150 max allowed)
9133797 hits, 8 misses, 24 trims, 24 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 1000, permanent 1000):
1000 in free list (5 min, 150 max allowed)
106010761 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 20, permanent 20, peak 21 @ 30w2d):
20 in free list (0 min, 100 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 25, permanent 25):
25 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 25, permanent 25):
25 in free list (0 min, 4 max allowed)
12831903 hits, 0 misses, 1 trims, 1 created
0 failures (0 no memory)

I'm looking for potential issues, tipps and suggestions for testing. A baseline config from a working pair would be helpful in case i am missing some tweaks possible.

A normal 10G-10G should be getting more throughput than 400-600M

I can outrule the notebooks as an issue. They have made proper measurements today on different devices.

Best Regards 

Chris

[Joseph W. Doherty]

"I can outrule the notebooks as an issue. They have made proper measurements today on different devices."

Please elaborate.

What are the all the port stats, end-to-end?

[Chris Rug]

Regarding the measurements:

Both notebooks have seen tests on speedtest servers showing close to 1G both directions properly. In general, they should be able to handle the traffic amount.

Here the interfaces and config on the interfaces. switch to switch. 

Side B

TenGigabitEthernet1/30 is up, line protocol is up 
Hardware is Ten Gigabit Ethernet Port, address is 78ba.f985.295d (bia 78ba.f985.295d)
Description: 10g-Test
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is force-up, media type is 10GBase-LR
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:14, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1904
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
15905125 packets input, 21191351825 bytes, 0 no buffer
Received 2006 broadcasts (1203 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
11536978 packets output, 12686026225 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Side A

TenGigabitEthernet1/15 is down, line protocol is down (notconnect)
Hardware is Ten Gigabit Ethernet Port, address is c4f7.d563.4c4e (bia c4f7.d563.4c4e)
Description: 10g-Test
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is force-up, media type is 10GBase-LR
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:01:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
7581143 packets input, 7783048376 bytes, 0 no buffer
Received 6552 broadcasts (4471 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
13120642 packets output, 17950599266 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
put packets with dribble condition detected
11536978 packets output, 12686026225 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Side A

interface TenGigabitEthernet1/30
description 10g-Test
switchport access vlan 334
switchport mode access
speed nonegotiate
end

Side B

interface TenGigabitEthernet1/15
description 10g-Test
switchport access vlan 334
switchport mode access
speed nonegotiate
end

The notebooks both have an interface using a SFP-GE-T SFP:

interface TenGigabitEthernet1/16
description 10g-Test
switchport access vlan 334
switchport mode access

interface TenGigabitEthernet1/29
description 10g-Test
switchport access vlan 334
switchport mode access

[Joseph W. Doherty · Accepted solution]

BTW, I asked for elaboration of how you tested your notebooks, earlier, to confirm you were sure they were not a factor in the tested performance being less than expected.  But, didn't think to ask, you've swapped the positions of the two notebooks and see the same performance results (i.e. throughput stats didn't also swap in direction)?

I also asked for all transit interface stats, not just those for two 10g interfaces, but that aside . . .

To confirm, the setup is, each notebook is connected to one of the 4500Xs on a 10g SFP+ port using a gig copper SFP, correct?

Your 10g ports show zero traffic rate at the time of the show interfaces taken?

I also notice flow-control is enabled and active.(?)

I see side B interface up but side A interface down?

I see two sets of some of the interface stats for the side A interface?  (The second set seems to be a repeat of side B.)

I see:

Side B
15905125 packets input,   21191351825 bytes, 0 no buffer
11536978 packets output, 12686026225 bytes, 0 underruns

Side A
  7581143 packets input,     7783048376 bytes, 0 no buffer
13120642 packets output, 17950599266 bytes, 0 underruns

The two side don't nearly match packets counts, but probably that has much to due with how long those counts were for on each 4500X.

However, those counts to show an asymmetrical traffic volume, both in packets and bytes, between the two sides.

Interestingly, the higher volume direction, A => B, might correspond with your lower test throughput direction too, but unsure as I'm not certain how your usage of upstream and downstream correspond with interface ingress/egress.  (Normally, upstream means toward the core and downstream toward the edge, but these 4500Xs are peers?)

Also BTW, as you OP was looking at buffers, your 4500Xs might support the buffers tune automatic command.  If so, are you using it?

[Chris Rug]

Dear all,

The issue has been handled by TAC and fixed after FW upgrade and cold reboot.

So thanks everybody for trying to help out.

I will rate the last reply as the best one.

Best Regards

C