08-06-2019 06:58 PM
Hello,
I've installed c5921 on a small Intel atom based device. I've configured the router with two DHCP service ports. I plugged a Linux server into one, and a laptop into the other. Running ethtool on both ports shows the negotiated speed of "1000baseT/Full".
Yet when I run iperf3 between the server and client, the throughput is almost non-existent (2Mb followed by zeros). When I attempt to download a large file (HTTP), it starts out reasonably, but very quickly (seconds) drops off to zero throughput.
Indications are that the 5921 is throttling the data, but I don't know why. How can I figure this out?
Solved! Go to Solution.
08-22-2019 03:57 AM
Analysis indicates that this problem is caused by network interface offloading. I use the following script on startup, though it can probably be simplified to just disable gro:
for int in $(ls -1 /sys/class/net/ | egrep "^en"); do ethtool -K $int rx off tx off sg off tso off gso off gro off done
I don't understand why everyone using the 5921 ESR doesn't have this problem.
08-07-2019 03:14 AM
Hello jbize,
your understanding is correct the ESR 5921 can perform bandwitdth throttling
see on pag .36 of the following pdf
use
show license
show platform license
to see what licenses are installed and in use on it. This is explained in previous pages of above document.
Hope to help
Giuseppe
08-07-2019 06:08 AM
Thanks for responding Giuseppe.
I've included the licensing information below.
From the startup:
*Aug 7 12:58:48.948: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/3, changed state to up *Aug 7 12:58:52.602: %SMART_LIC-5-EVAL_START: Entering evaluation period *Aug 7 12:58:52.602: %LICENSE_C5920-6-LICENSE_ACTIVATED: Installed license for feature c5921-x86-level6 now in use. Forwarding bandwidth limited to 500 Mbps *Aug 7 12:58:53.820: %SYS-5-CONFIG_I: Configured from memory by console *Aug 7 12:58:53.820: %SMART_LIC-6-HA_ROLE_CHANGED: Smart Agent HA role changed to Active. *Aug 7 12:58:53.890: %SYS-5-RESTART: System restarted -- Cisco IOS Software, C5921 Software (C5921_I86-UNIVERSALK9-M), Version 15.7(3)M4, RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2019 by Cisco Systems, Inc. Compiled Sat 09-Feb-19 13:33 by prod_rel_team *Aug 7 12:58:54.043: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF *Aug 7 12:58:54.043: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF *Aug 7 12:58:54.554: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
Router#show license % Incomplete command. Router#show license all Smart Licensing Status ====================== Smart Licensing is ENABLED Registration: Status: UNREGISTERED - REGISTRATION FAILED Export-Controlled Functionality: Not Allowed Initial Registration: FAILED on Aug 02 13:54:02 2019 EST Failure reason: Fail to send out Call Home HTTP message. License Authorization: Status: EVAL MODE Evaluation Period Remaining: 89 days, 23 hours, 51 minutes, 43 seconds License Usage ============== (c5921-x86-level6): Description: Count: 1 Version: 1.0 Status: EVAL MODE Product Information =================== UDI: PID:CISCO5921-K9,SN:9GS3158OB4O Agent Version ============= Smart Agent for Licensing: 1.6.10_rel/106 Component Versions: SA:(1_6_rel)3.2.18, SI:(rel22)1.5.5, CH:(rel4)2.0.1, PK:(rel18)1.1.87
Router#show platform software license Packet forwarding: Enabled Current enforcement forwarding rate: 500 Mbps Unique Device Identifier: CISCO5921-K9:9GS3158OB4O License features supported: Feature Rate Status --------------------- --------- ------ c5921-x86-default 8 Kbps - c5921-x86-evaluation 50 Mbps - c5921-x86-level0 5 Mbps - c5921-x86-level1 10 Mbps - c5921-x86-level2 25 Mbps - c5921-x86-level3 50 Mbps - c5921-x86-level4 100 Mbps - c5921-x86-level5 200 Mbps - c5921-x86-level6 500 Mbps In Use
08-07-2019 06:46 AM
BTW, This is a typical iperf3 output:
C:\Progs\iperf-3.1.3-win64>iperf3.exe -c 192.168.132.52 Connecting to host 192.168.132.52, port 5201 [ 4] local 192.168.32.52 port 1735 connected to 192.168.132.52 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 256 KBytes 2.07 Mbits/sec [ 4] 1.01-2.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 2.01-3.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 3.01-4.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 4.01-5.00 sec 0.00 Bytes 0.00 bits/sec [ 4] 5.00-6.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 6.01-7.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 7.01-8.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 8.01-9.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 9.01-10.01 sec 0.00 Bytes 0.00 bits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.01 sec 256 KBytes 210 Kbits/sec sender [ 4] 0.00-10.01 sec 52.8 KBytes 43.2 Kbits/sec receiver iperf Done.
It has occasionally started out a little better, but always settles on this.
08-09-2019 12:56 PM
I have a significant update...
Out of frustration, I tried switching from 32Bit Debian9 to 64Bit CentOS7. I was working on a very kludged up device I pasted together with a minnowboard a USB2 hub and a handful of USB to Ethernet adapters. Working with the same test config, 2 DHCP clients on two adjacent ports, and I started getting a consistent 9.44 Mbps iperf3 rate. Checking with ethtool (and shpw platform software interface-status), I saw that one of my ports was being configured to 10Mbps instead of 1000. Well, I migrated one of my real devices over to the CentOS7 64, and things went back to working as before with Debian9.
It turns out that as long as one of the ports I'm using is configured for a speed of 10Mbps, I get a 10Mbps throughput. However, if all of the ports being used are 100 or 1000, I get a small burst followed by nothing (see previous post).
What is going on?
08-10-2019 06:54 AM - edited 08-10-2019 07:01 AM
Hello,
>>
*Aug 7 12:58:52.602: %LICENSE_C5920-6-LICENSE_ACTIVATED: Installed license for feature c5921-x86-level6 now in use. Forwarding bandwidth limited to 500 Mbps
I agree you should get far better results and consistent over time.
I am not able to help you on the possible issues.
Verify if the hardware can support this SW appliance and if you need to tweak the processor bios settings.
Edit:
You have issues with the underlying hardware and os as you have noted in your last post.
There might be some tuning to be performed at BIOS and linux level but I am not able to help on these aspects.
IS the processor still an intel ATom box ?
Hope to help
Giuseppe
08-10-2019 12:26 PM - edited 08-10-2019 12:30 PM
Yes, the processors used are Intel Atom dual and quad core. I don't see any performance issues in Linux (outside the 5921). I have tried several different configurations, all with the same results.
I have confirmed this behavior with Intel, Realtek, Broadcom, and more network interfaces (3-4 different drivers and versions). The Debian 9 (32 bit) Kernel version is 4.9.0-9. The CentOS7 (64 bit) kernel version is 3.10.
The c5921 distribution I am using is: c5921i86-universalk9-tar.SPA.157-3.M4.tar
The port configuration settings in SWROPTIONS are promiscuous, monitor-state, and push-mon-int all true. The Linux NetworkManager/networking service is off.
What can I do do isolate the issue? What logging/debugging should I enable. What parameters (bandwidth, rate-limit, etc.) can I modify to affect results. Is there a setting (or settings) that would "open up" any limiting, filter, shaping, etc. that may be in effect by default?
Thanks for your help.
08-11-2019 05:12 AM
Hello jbize,
according to the pdf document I have linked in my first answer in this thread:
on pag 10-11/67
The Cisco5921ESR has the following hardware requirements:
•The host platform must be Intel x86 or compatible family with a minimum 512 Mbytes of RAM
The Linux kernel must be version 2.6.32 or greater.
•The Cisco 5921 ESR is a 32-bit application. It will work on a 64-bit Linux system, provided you install and use the 32-bit libraries.
•Standard Linux packages installed on the Linux system.
•The Cisco 5921 ESR has no Linux distribution restrictions. You are responsible for building and verifying functionality. Cisco has tested the Cisco 5921 ESR with CentOS using desktop and embedded configurations
The Cisco5921ESR is contained in a compiled tar file with the following components:
•c5921i86-universalk9-ms—application that contains the Cisco IOS code.
•swrvcon—Linux application that provides a virtual console to access Cisco IOS
•swr_reload—Used to launch and reload the software router in the event that the underlaying Linux OS develops any problems.
•libdyncs.so—Application-specific shared library.
•Man pages for the Cisco 5921 ESR applications/files: swr-application, swr_reload, swroptions, and swrvcon.
If all the SW components listed above are installed you should be fine.
>> The Linux NetworkManager/networking service is off.
I don't know if this is a required step. I wonder if this might be an issue. What happens if you try to enable the linux networking service ?
Also have you installed the 32 bit libraries when trying to use it on 64 bit linux o.s.?
Unfortunately I am not able to provide you more help.
What if you try to use the swr_reload to restart the C5921 SW appliance ?
Also the issues may be in the drivers as you are using USB to ethernet adapters.
The requirements look like not so challenging.
I would suggest to try one change per time and observe the results :
a) enabling networking service on linux
b) using the swr_reload option
c) considering to move the SW appliance to a server with two native Ethernet NICs.
Hope to help
Giuseppe
08-11-2019 07:01 AM
Hello Giuseppe,
I don't think I was clear in my previous post. I've mostly tried to use the router with Debian 9 32bit Linux and native Ethernet interfaces, both Intel and Broadcom. Without using the c5921 software, the throughput is fine. When I temporarily switched over to CentOS7 64bit for testing, I did use the required 32bit libraries. I use swr_reload every time I start or stop the router (via the shell script).
The requirements look like not so challenging.
I agree.
I would suggest to try one change per time and observe the results :
a) enabling networking service on linux
Yes, I only recently turned it off to take it out of the equation.
b) using the swr_reload option
Yes, that is the start/stop control application.
c) considering to move the SW appliance to a server with two native Ethernet NICs.
Yes, that it the normal setup. I used the other devices to try something different.
What seems consistent across all the hardware and software stacks I've tried, is it works well at 10Mbps, but not at 100 or 1000.
Can you tell me how to diagnose any bandwidth or rate-limiting issues in the router? How do I enable the kind of debugging or logging that would indicate how the traffic is being restricted? How can I "open up" any bandwidth, shaping, or rate-limiting restrictions?
Thanks,
John
08-11-2019 07:47 AM
Hello John,
to check if the ESR5921 is dropping traffic you could start by looking at
show interfaces
output looking for input errors like overruns , ignored or output drops in traditional routers is the first sign of a performance issue or of the action of a QoS policy-map.
However, your iperf tests show that in the first minute of traffic you get something and then almost nothing.
So I am afraid there is some issue on the way this SW appliance runs over Linux.
Any type of performance bottleneck on the SW appliance itself should provide you the same performance over time. Here the results fall down to zero after one minute.
To check the configuration you can attach a file with show run or show conf and also a show interface.
Hope to help
Giuseppe
08-11-2019 08:03 AM - edited 08-11-2019 08:04 AM
Thanks Giuseppe. I will do that when I get in to the office tomorrow.
PS: It's not the "first minute," but the first second.
08-12-2019 08:14 AM - edited 08-12-2019 08:19 AM
Here is a full run configuration, license info, and test results:
Router# Router#show license status Smart Licensing is ENABLED Registration: Status: UNREGISTERED - REGISTRATION FAILED Export-Controlled Functionality: Not Allowed Initial Registration: FAILED on Aug 02 14:13:57 2019 EST Failure reason: Fail to send out Call Home HTTP message. License Authorization: Status: EVAL MODE Evaluation Period Remaining: 89 days, 23 hours, 28 minutes, 13 seconds Router# Router#show platform software license Packet forwarding: Enabled Current enforcement forwarding rate: 50 Mbps Unique Device Identifier: CISCO5921-K9:9GS3158OB4O License features supported: Feature Rate Status --------------------- --------- ------ c5921-x86-default 8 Kbps - c5921-x86-evaluation 50 Mbps - c5921-x86-level0 5 Mbps - c5921-x86-level1 10 Mbps - c5921-x86-level2 25 Mbps - c5921-x86-level3 50 Mbps In Use c5921-x86-level4 100 Mbps - c5921-x86-level5 200 Mbps - c5921-x86-level6 500 Mbps - Router# Router# Router#show run Building configuration... Current configuration : 4398 bytes ! ! Last configuration change at 10:03:14 EST Mon Aug 12 2019 ! version 15.7 service timestamps debug datetime msec service timestamps log datetime msec no service password-encryption ! Call-home is enabled by Smart-Licensing. service call-home ! hostname Router ! boot-start-marker boot-end-marker ! ! ! no aaa new-model clock timezone EST -5 0 mmi polling-interval 60 no mmi auto-configure no mmi pvc mmi snmp-timeout 180 call-home ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications. contact-email-addr sch-smart-licensing@cisco.com profile "CiscoTAC-1" active destination transport-method http no destination transport-method email ! ! ! ! ! ! ! ! ! ! ! ip dhcp excluded-address 192.168.32.1 192.168.32.50 ip dhcp excluded-address 192.168.132.1 192.168.132.50 ! ip dhcp pool tempDHCP network 192.168.32.0 255.255.255.0 default-router 192.168.32.1 ! ip dhcp pool temp2DHCP network 192.168.132.0 255.255.255.0 default-router 192.168.132.1 ! ! ! no ip domain lookup ip cef no ipv6 cef ! multilink bundle-name authenticated ! ! ! ! ! ! ! ! ! crypto pki trustpoint SLA-TrustPoint enrollment pkcs12 revocation-check crl ! ! crypto pki certificate chain SLA-TrustPoint certificate ca 01 30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030 32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934 3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720 526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030 82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520 1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE 4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC 7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188 68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7 C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191 C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44 DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201 06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85 4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500 03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905 604EDCDE FF4FED2B 77FC460E CD636FDB DD44681E 3A5673AB 9093D3B1 6C9E3D8B D98987BF E40CBD9E 1AECA0C2 2189BB5C 8FA85686 CD98B646 5575B146 8DFC66A8 467A3DF4 4D565700 6ADF0F0D CF835015 3C04FF7C 21E878AC 11BA9CD2 55A9232C 7CA7B7E6 C1AF74F6 152E99B7 B1FCF9BB E973DE7F 5BDDEB86 C71E3B49 1765308B 5FB0DA06 B92AFE7F 494E8A9E 07B85737 F3A58BE1 1A48A229 C37C1E69 39F08678 80DDCD16 D6BACECA EEBC7CF9 8428787B 35202CDC 60E4616A B623CDBD 230E3AFB 418616A9 4093E049 4D10AB75 27E86F73 932E35B5 8862FDAE 0275156F 719BB2F0 D697DF7F 28 quit license udi pid CISCO5921-K9 sn 9GS3158OB4O license platform throughput level c5921-x86-level3 license smart enable ! redundancy ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! interface Ethernet0/0 ip address dhcp duplex auto speed auto ! interface Ethernet0/1 no ip address shutdown ! interface Ethernet0/2 no ip address shutdown ! interface Ethernet0/3 no ip address shutdown ! interface Ethernet1/0 ip address 192.168.32.1 255.255.255.0 ip nat inside ip virtual-reassembly in duplex auto speed auto ! interface Ethernet1/1 ip address 192.168.132.1 255.255.255.0 ip nat inside ip virtual-reassembly in duplex auto speed auto ! interface Ethernet1/2 no ip address shutdown duplex auto speed auto ! interface Ethernet1/3 no ip address shutdown duplex auto speed auto ! ip forward-protocol nd ! ! no ip http server no ip http secure-server ! ipv6 ioam timestamp ! ! ! control-plane ! ! ! ! ! ! ! ! ! ! ! alias exec s show ip interface brief alias exec sr show ip route ! line con 0 exec-timeout 0 0 logging synchronous line aux 0 line vty 0 4 login transport input none ! ntp server 34.202.215.187 ! ! ! ! ! ! ! end Router# Router# Router#show interface Ethernet0/0 is up, line protocol is up Hardware is AmdP2, address is 40d8.550d.100e (bia 40d8.550d.100e) Internet address is 192.168.5.119/24 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:08, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 4909 packets input, 635508 bytes, 0 no buffer Received 4531 broadcasts (145 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 520 packets output, 62200 bytes, 0 underruns 1 output errors, 1 collisions, 1 interface resets 934 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Ethernet0/1 is administratively down, line protocol is down Hardware is AmdP2, address is aa47.5333.3110 (bia aa47.5333.3110) MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Ethernet0/2 is administratively down, line protocol is down Hardware is AmdP2, address is aa47.5333.3120 (bia aa47.5333.3120) MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Ethernet0/3 is administratively down, line protocol is down Hardware is AmdP2, address is aa47.5333.3130 (bia aa47.5333.3130) MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Ethernet1/0 is up, line protocol is up Hardware is AmdP2, address is 40d8.550d.100a (bia 40d8.550d.100a) Internet address is 192.168.32.1/24 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:15, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/57/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 355000 bits/sec, 22 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 34894 packets input, 47373920 bytes, 0 no buffer Received 3127 broadcasts (128 IP multicasts) 0 runts, 257 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 17422 packets output, 1028454 bytes, 0 underruns 1 output errors, 1 collisions, 2 interface resets 1337 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Ethernet1/1 is up, line protocol is up Hardware is AmdP2, address is 40d8.550d.100b (bia 40d8.550d.100b) Internet address is 192.168.132.1/24 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:10, output 00:00:01, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 352000 bits/sec, 18 packets/sec 17343 packets input, 1087465 bytes, 0 no buffer Received 135 broadcasts (37 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 31212 packets output, 46742688 bytes, 0 underruns 1 output errors, 1 collisions, 2 interface resets 69 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Ethernet1/2 is administratively down, line protocol is down Hardware is AmdP2, address is 40d8.550d.100c (bia 40d8.550d.100c) MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Ethernet1/3 is administratively down, line protocol is down Hardware is AmdP2, address is 40d8.550d.100d (bia 40d8.550d.100d) MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Half-duplex ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out NVI0 is up, line protocol is up Hardware is NVI Interface is unnumbered. Using address of Ethernet0/0 (192.168.5.119) MTU 1514 bytes, BW 56 Kbit/sec, DLY 5000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation UNKNOWN, loopback not set Keepalive set (10 sec) Unknown duplex Last input never, output never, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out Router# Router# Router#show platform software interface-status Status of Monitored Interfaces ------------------------------ Linux Interface IOS Running Carrier MTU Speed Duplex Autoneg enp1s0f3 1/3 No No 1500 ??? ??? Yes enp1s0f2 1/2 No No 1500 ??? ??? Yes enp1s0f1 1/1 Yes Yes 1500 1 Gbps full Yes enp1s0f0 1/0 Yes Yes 1500 100 Mbps full Yes enp3s0 0/0 Yes Yes 1500 1 Gbps full Yes Router# Router# Router#!!! Captured first iperf3... Router#
First test results:
C:\Progs\iperf-3.1.3-win64>iperf3.exe -c 192.168.132.52 Connecting to host 192.168.132.52, port 5201 [ 4] local 192.168.32.52 port 25203 connected to 192.168.132.52 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 1.01-2.00 sec 0.00 Bytes 0.00 bits/sec [ 4] 2.00-3.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 3.01-4.00 sec 0.00 Bytes 0.00 bits/sec [ 4] 4.00-5.01 sec 0.00 Bytes 0.00 bits/sec [ 4] 5.01-6.01 sec 128 KBytes 1.05 Mbits/sec [ 4] 6.01-7.00 sec 512 KBytes 4.24 Mbits/sec [ 4] 7.00-8.00 sec 768 KBytes 6.30 Mbits/sec [ 4] 8.00-9.00 sec 768 KBytes 6.28 Mbits/sec [ 4] 9.00-10.00 sec 896 KBytes 7.34 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 3.25 MBytes 2.73 Mbits/sec sender [ 4] 0.00-10.00 sec 3.05 MBytes 2.56 Mbits/sec receiver iperf Done.
I change one of the ports to 10Mbps:
Router# Router# Router#conf t Enter configuration commands, one per line. End with CNTL/Z. Router(config)#int e1/0 Router(config-if)#speed 10 Router(config-if)# Aug 12 15:06:38.074: %LINK-3-UPDOWN: Interface Ethernet1/0, changed state to down Aug 12 15:06:39.075: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to down Router(config-if)# Aug 12 15:06:41.216: %LINK-3-UPDOWN: Interface Ethernet1/0, changed state to up Aug 12 15:06:42.216: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to up Router(config-if)# Router(config-if)# Router(config-if)# Router(config-if)#!!! Second iperf3... Router(config-if)#
And run the second test:
C:\Progs\iperf-3.1.3-win64>iperf3.exe -c 192.168.132.52 Connecting to host 192.168.132.52, port 5201 [ 4] local 192.168.32.52 port 25206 connected to 192.168.132.52 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 1.38 MBytes 11.4 Mbits/sec [ 4] 1.01-2.01 sec 1.12 MBytes 9.47 Mbits/sec [ 4] 2.01-3.01 sec 1.12 MBytes 9.40 Mbits/sec [ 4] 3.01-4.01 sec 1.12 MBytes 9.44 Mbits/sec [ 4] 4.01-5.01 sec 1.12 MBytes 9.42 Mbits/sec [ 4] 5.01-6.01 sec 1.12 MBytes 9.44 Mbits/sec [ 4] 6.01-7.01 sec 1.12 MBytes 9.44 Mbits/sec [ 4] 7.01-8.01 sec 1.12 MBytes 9.43 Mbits/sec [ 4] 8.01-9.01 sec 1.12 MBytes 9.45 Mbits/sec [ 4] 9.01-10.01 sec 1.12 MBytes 9.47 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.01 sec 11.5 MBytes 9.64 Mbits/sec sender [ 4] 0.00-10.01 sec 11.5 MBytes 9.64 Mbits/sec receiver iperf Done.
08-22-2019 03:57 AM
Analysis indicates that this problem is caused by network interface offloading. I use the following script on startup, though it can probably be simplified to just disable gro:
for int in $(ls -1 /sys/class/net/ | egrep "^en"); do ethtool -K $int rx off tx off sg off tso off gso off gro off done
I don't understand why everyone using the 5921 ESR doesn't have this problem.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide