cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2593
Views
5
Helpful
12
Replies

C5921 ESR performance

jbize
Level 1
Level 1

Hello,

I've installed c5921 on a small Intel atom based device.  I've configured the router with two DHCP service ports.  I plugged a Linux server into one, and a laptop into the other.  Running ethtool on both ports shows the negotiated speed of "1000baseT/Full".

Yet when I run iperf3 between the server and client, the throughput is almost non-existent (2Mb followed by zeros).  When I attempt to download a large file (HTTP), it starts out reasonably, but very quickly (seconds) drops off to zero throughput.

 

Indications are that the 5921 is throttling the data, but I don't know why.  How can I figure this out?

1 Accepted Solution

Accepted Solutions

Analysis indicates that this problem is caused by network interface offloading.  I use the following script on startup, though it can probably be simplified to just disable gro:

for int in $(ls -1 /sys/class/net/ | egrep "^en"); do
    ethtool -K $int rx off tx off sg off tso off gso off gro off
done

I don't understand why everyone using the 5921 ESR doesn't have this problem.

View solution in original post

12 Replies 12

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello jbize,

your understanding is correct the ESR 5921 can perform bandwitdth throttling

see on pag .36 of the following pdf

 

https://www.cisco.com/c/dam/en/us/td/docs/solutions/GGSG-Engineering/Cisco_5921/Cisco_5921_ESR_Integration.pdf

 

 

use

show license

show platform license

 

to see what licenses are installed and in use on it. This is explained in previous pages of above document.

 

Hope to help

Giuseppe

 

Thanks for responding Giuseppe. 

 

I've included the licensing information below.

 

From the startup:

*Aug  7 12:58:48.948: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/3, changed state to up
*Aug  7 12:58:52.602: %SMART_LIC-5-EVAL_START: Entering evaluation period
*Aug  7 12:58:52.602: %LICENSE_C5920-6-LICENSE_ACTIVATED: Installed license for feature c5921-x86-level6 now in use. Forwarding bandwidth limited to 500 Mbps
*Aug  7 12:58:53.820: %SYS-5-CONFIG_I: Configured from memory by console
*Aug  7 12:58:53.820: %SMART_LIC-6-HA_ROLE_CHANGED: Smart Agent HA role changed to Active.
*Aug  7 12:58:53.890: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C5921 Software (C5921_I86-UNIVERSALK9-M), Version 15.7(3)M4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Sat 09-Feb-19 13:33 by prod_rel_team
*Aug  7 12:58:54.043: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Aug  7 12:58:54.043: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Aug  7 12:58:54.554: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
Router#show license
% Incomplete command.

Router#show license all
Smart Licensing Status
======================

Smart Licensing is ENABLED

Registration:
  Status: UNREGISTERED - REGISTRATION FAILED
  Export-Controlled Functionality: Not Allowed
  Initial Registration: FAILED on Aug 02 13:54:02 2019 EST
    Failure reason: Fail to send out Call Home HTTP message.

License Authorization:
  Status: EVAL MODE
  Evaluation Period Remaining: 89 days, 23 hours, 51 minutes, 43 seconds

License Usage
==============

(c5921-x86-level6):
  Description:
  Count: 1
  Version: 1.0
  Status: EVAL MODE

Product Information
===================
UDI: PID:CISCO5921-K9,SN:9GS3158OB4O

Agent Version
=============
Smart Agent for Licensing: 1.6.10_rel/106
Component Versions: SA:(1_6_rel)3.2.18, SI:(rel22)1.5.5, CH:(rel4)2.0.1, PK:(rel18)1.1.87

 

Router#show platform software license

Packet forwarding: Enabled

Current enforcement forwarding rate: 500 Mbps

Unique Device Identifier: CISCO5921-K9:9GS3158OB4O

License features supported:
       Feature              Rate     Status
  ---------------------   ---------  ------
  c5921-x86-default          8 Kbps    -
  c5921-x86-evaluation      50 Mbps    -
  c5921-x86-level0           5 Mbps    -
  c5921-x86-level1          10 Mbps    -
  c5921-x86-level2          25 Mbps    -
  c5921-x86-level3          50 Mbps    -
  c5921-x86-level4         100 Mbps    -
  c5921-x86-level5         200 Mbps    -
  c5921-x86-level6         500 Mbps  In Use

BTW,  This is a typical iperf3 output:

C:\Progs\iperf-3.1.3-win64>iperf3.exe -c 192.168.132.52
Connecting to host 192.168.132.52, port 5201
[  4] local 192.168.32.52 port 1735 connected to 192.168.132.52 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   256 KBytes  2.07 Mbits/sec
[  4]   1.01-2.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.01-4.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   4.01-5.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   5.00-6.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   6.01-7.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   7.01-8.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   8.01-9.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   256 KBytes   210 Kbits/sec                  sender
[  4]   0.00-10.01  sec  52.8 KBytes  43.2 Kbits/sec                  receiver

iperf Done.

It has occasionally started out a little better, but always settles on this.

I have a significant update...

Out of frustration, I tried switching from 32Bit Debian9 to 64Bit CentOS7.  I was working on a very kludged up device I pasted together with a minnowboard a USB2 hub and a handful of USB to Ethernet adapters.  Working with the same test config, 2 DHCP clients on two adjacent ports, and I started getting a consistent 9.44 Mbps iperf3 rate.  Checking with ethtool (and shpw platform software interface-status), I saw that one of my ports was being configured to 10Mbps instead of 1000.  Well, I migrated one of my real devices over to the CentOS7 64, and things went back to working as before with Debian9.

 

It turns out that as long as one of the ports I'm using is configured for a speed of 10Mbps, I get a 10Mbps throughput.  However, if all of the ports being used are 100 or 1000, I get a small burst followed by nothing (see previous post).

 

What is going on?

 

Hello,

 

>>

*Aug  7 12:58:52.602: %LICENSE_C5920-6-LICENSE_ACTIVATED: Installed license for feature c5921-x86-level6 now in use. Forwarding bandwidth limited to 500 Mbps

I agree you should get far better results and consistent over time.

 I am not able to help you on the possible issues.

Verify if the hardware can support this SW appliance and if you need to tweak the processor bios settings.

 

Edit:

You have issues with the underlying hardware and os as you have noted in your last post.

There might be some tuning to be performed at BIOS and linux level but I am not able to help on these aspects.

IS the processor still an intel ATom box ?

 

Hope to help

Giuseppe

 

Yes, the processors used are Intel Atom dual and quad core. I don't see any performance issues in Linux (outside the 5921).  I have tried several different configurations, all with the same results.

I have confirmed this behavior with Intel, Realtek, Broadcom, and more network interfaces (3-4 different drivers and versions).  The Debian 9 (32 bit) Kernel version is 4.9.0-9.  The CentOS7 (64 bit) kernel version is 3.10.

The c5921 distribution I am using is: c5921i86-universalk9-tar.SPA.157-3.M4.tar

The port configuration settings in SWROPTIONS are promiscuous, monitor-state, and push-mon-int all true.  The Linux NetworkManager/networking service is off.

 

What can I do do isolate the issue?  What logging/debugging should I enable.  What parameters (bandwidth, rate-limit, etc.) can I modify to affect results.  Is there a setting (or settings) that would "open up" any limiting, filter, shaping, etc. that may be in effect by default?

 

Thanks for your help.

Hello jbize,

according to the pdf document I have linked in my first answer in this thread:

on pag 10-11/67

The Cisco5921ESR has the following hardware requirements:

The host platform must be Intel x86 or compatible family with a minimum 512 Mbytes of RAM

The Linux kernel must be version 2.6.32 or greater.

The Cisco 5921 ESR is a 32-bit application. It will work on a 64-bit Linux system, provided you install and use the 32-bit libraries.

Standard Linux packages installed on the Linux system.

The Cisco 5921 ESR has no Linux distribution restrictions. You are responsible for building and verifying functionality. Cisco has tested the Cisco 5921 ESR with CentOS using desktop and embedded configurations

 

The Cisco5921ESR is contained in a compiled tar file with the following components:

c5921i86-universalk9-ms—application that contains the Cisco IOS code.

swrvcon—Linux application that provides a virtual console to access Cisco IOS

swr_reload—Used to launch and reload the software router in the event that the underlaying Linux OS develops any problems.

libdyncs.so—Application-specific shared library.

Man pages for the Cisco 5921 ESR applications/files: swr-application, swr_reload, swroptions, and swrvcon.

 

If all the SW components listed above are installed you should be fine.

 

>> The Linux NetworkManager/networking service is off.

 

I don't know if this is a required step. I wonder if this might be an issue. What happens if you try to enable the linux networking service ?

Also have you installed the 32 bit libraries when trying to use it on 64 bit linux o.s.?

 

Unfortunately I am not able to provide you more help.

What if you try to use the swr_reload to restart the C5921 SW appliance ?

 

Also the issues may be in the drivers as you are using USB to ethernet adapters.

 

The requirements look like not so challenging.

I would suggest to try one change per time and observe the results :

a) enabling networking service on linux

b) using the swr_reload option

c) considering to move the SW appliance to a server with two native Ethernet NICs.

 

Hope to help

Giuseppe

 

Hello Giuseppe,

 

I don't think I was clear in my previous post.  I've mostly tried to use the router with Debian 9 32bit Linux and native Ethernet interfaces, both Intel and Broadcom.  Without using the c5921 software, the throughput is fine.  When I temporarily switched over to CentOS7 64bit for testing, I did use the required 32bit libraries.  I use swr_reload every time I start or stop the router (via the shell script).

 

The requirements look like not so challenging.

    I agree.

I would suggest to try one change per time and observe the results :

a) enabling networking service on linux

    Yes, I only recently turned it off to take it out of the equation.

b) using the swr_reload option

    Yes, that is the start/stop control application.

c) considering to move the SW appliance to a server with two native Ethernet NICs.

    Yes, that it the normal setup.  I used the other devices to try something different.

 

What seems consistent across all the hardware and software stacks I've tried, is it works well at 10Mbps, but not at 100 or 1000.

 

Can you tell me how to diagnose any bandwidth or rate-limiting issues in the router?  How do I enable the kind of debugging or logging that would indicate how the traffic is being restricted?  How can I "open up" any bandwidth, shaping, or rate-limiting restrictions?

 

Thanks,

John

Hello John,

to check if the ESR5921 is dropping traffic you could start by looking at

show interfaces

output looking for input errors like overruns , ignored or output drops in traditional routers is the first sign of a performance issue or of the action of a QoS policy-map.

 

However, your iperf tests show that in the first minute of traffic you get something and then almost nothing.

So I am afraid there is some issue on the way this SW appliance runs over Linux.

Any type of performance bottleneck on the SW appliance itself should provide you the same performance over time. Here the results fall down to zero after one minute.

 

To check the configuration you can attach a file with show run or show conf and also a show interface.

 

Hope to help

Giuseppe

 

Thanks Giuseppe.  I will do that when I get in to the office tomorrow.

 

PS: It's not the "first minute," but the first second.

Here is a full run configuration, license info, and test results:

Router#
Router#show license status
Smart Licensing is ENABLED

Registration:
  Status: UNREGISTERED - REGISTRATION FAILED
  Export-Controlled Functionality: Not Allowed
  Initial Registration: FAILED on Aug 02 14:13:57 2019 EST
    Failure reason: Fail to send out Call Home HTTP message.

License Authorization:
  Status: EVAL MODE
  Evaluation Period Remaining: 89 days, 23 hours, 28 minutes, 13 seconds


Router#
Router#show platform software license

Packet forwarding: Enabled

Current enforcement forwarding rate: 50 Mbps

Unique Device Identifier: CISCO5921-K9:9GS3158OB4O

License features supported:
       Feature              Rate     Status
  ---------------------   ---------  ------
  c5921-x86-default          8 Kbps    -
  c5921-x86-evaluation      50 Mbps    -
  c5921-x86-level0           5 Mbps    -
  c5921-x86-level1          10 Mbps    -
  c5921-x86-level2          25 Mbps    -
  c5921-x86-level3          50 Mbps  In Use
  c5921-x86-level4         100 Mbps    -
  c5921-x86-level5         200 Mbps    -
  c5921-x86-level6         500 Mbps    -

Router#
Router#
Router#show run
Building configuration...

Current configuration : 4398 bytes
!
! Last configuration change at 10:03:14 EST Mon Aug 12 2019
!
version 15.7
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
! Call-home is enabled by Smart-Licensing.
service call-home
!
hostname Router
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
clock timezone EST -5 0
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
call-home
 ! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
 ! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
 contact-email-addr sch-smart-licensing@cisco.com
 profile "CiscoTAC-1"
  active
  destination transport-method http
  no destination transport-method email
!
!
!
!
!
!
!
!
!
!


!
ip dhcp excluded-address 192.168.32.1 192.168.32.50
ip dhcp excluded-address 192.168.132.1 192.168.132.50
!
ip dhcp pool tempDHCP
 network 192.168.32.0 255.255.255.0
 default-router 192.168.32.1
!
ip dhcp pool temp2DHCP
 network 192.168.132.0 255.255.255.0
 default-router 192.168.132.1
!
!
!
no ip domain lookup
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
crypto pki trustpoint SLA-TrustPoint
 enrollment pkcs12
 revocation-check crl
!
!
crypto pki certificate chain SLA-TrustPoint
 certificate ca 01
  30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030
  32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363
  6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934
  3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305
  43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720
  526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030
  82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D
  CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520
  1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE
  4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC
  7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188
  68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7
  C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191
  C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44
  DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201
  06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85
  4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500
  03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905
  604EDCDE FF4FED2B 77FC460E CD636FDB DD44681E 3A5673AB 9093D3B1 6C9E3D8B
  D98987BF E40CBD9E 1AECA0C2 2189BB5C 8FA85686 CD98B646 5575B146 8DFC66A8
  467A3DF4 4D565700 6ADF0F0D CF835015 3C04FF7C 21E878AC 11BA9CD2 55A9232C
  7CA7B7E6 C1AF74F6 152E99B7 B1FCF9BB E973DE7F 5BDDEB86 C71E3B49 1765308B
  5FB0DA06 B92AFE7F 494E8A9E 07B85737 F3A58BE1 1A48A229 C37C1E69 39F08678
  80DDCD16 D6BACECA EEBC7CF9 8428787B 35202CDC 60E4616A B623CDBD 230E3AFB
  418616A9 4093E049 4D10AB75 27E86F73 932E35B5 8862FDAE 0275156F 719BB2F0
  D697DF7F 28
        quit
license udi pid CISCO5921-K9 sn 9GS3158OB4O
license platform throughput level c5921-x86-level3
license smart enable
!
redundancy

!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Ethernet0/0
 ip address dhcp
 duplex auto
 speed auto
!
interface Ethernet0/1
 no ip address
 shutdown
!
interface Ethernet0/2
 no ip address
 shutdown
!
interface Ethernet0/3
 no ip address
 shutdown
!
interface Ethernet1/0
 ip address 192.168.32.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface Ethernet1/1
 ip address 192.168.132.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface Ethernet1/2
 no ip address
 shutdown
 duplex auto
 speed auto
!
interface Ethernet1/3
 no ip address
 shutdown
 duplex auto
 speed auto
!
ip forward-protocol nd
!
!
no ip http server
no ip http secure-server
!
ipv6 ioam timestamp
!
!
!
control-plane
!
 !
 !
 !
 !
!
!
!
!
!
!
alias exec s show ip interface brief
alias exec sr show ip route
!
line con 0
 exec-timeout 0 0
 logging synchronous
line aux 0
line vty 0 4
 login
 transport input none
!
ntp server 34.202.215.187
!
!
!
!
!
!
!
end

Router#
Router#
Router#show interface
Ethernet0/0 is up, line protocol is up
  Hardware is AmdP2, address is 40d8.550d.100e (bia 40d8.550d.100e)
  Internet address is 192.168.5.119/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:08, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 2 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     4909 packets input, 635508 bytes, 0 no buffer
     Received 4531 broadcasts (145 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     520 packets output, 62200 bytes, 0 underruns
     1 output errors, 1 collisions, 1 interface resets
     934 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Ethernet0/1 is administratively down, line protocol is down
  Hardware is AmdP2, address is aa47.5333.3110 (bia aa47.5333.3110)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Half-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Ethernet0/2 is administratively down, line protocol is down
  Hardware is AmdP2, address is aa47.5333.3120 (bia aa47.5333.3120)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Half-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Ethernet0/3 is administratively down, line protocol is down
  Hardware is AmdP2, address is aa47.5333.3130 (bia aa47.5333.3130)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Half-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Ethernet1/0 is up, line protocol is up
  Hardware is AmdP2, address is 40d8.550d.100a (bia 40d8.550d.100a)
  Internet address is 192.168.32.1/24
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:15, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/57/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 355000 bits/sec, 22 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     34894 packets input, 47373920 bytes, 0 no buffer
     Received 3127 broadcasts (128 IP multicasts)
     0 runts, 257 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     17422 packets output, 1028454 bytes, 0 underruns
     1 output errors, 1 collisions, 2 interface resets
     1337 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Ethernet1/1 is up, line protocol is up
  Hardware is AmdP2, address is 40d8.550d.100b (bia 40d8.550d.100b)
  Internet address is 192.168.132.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:10, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 352000 bits/sec, 18 packets/sec
     17343 packets input, 1087465 bytes, 0 no buffer
     Received 135 broadcasts (37 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     31212 packets output, 46742688 bytes, 0 underruns
     1 output errors, 1 collisions, 2 interface resets
     69 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Ethernet1/2 is administratively down, line protocol is down
  Hardware is AmdP2, address is 40d8.550d.100c (bia 40d8.550d.100c)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Half-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Ethernet1/3 is administratively down, line protocol is down
  Hardware is AmdP2, address is 40d8.550d.100d (bia 40d8.550d.100d)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Half-duplex
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 input packets with dribble condition detected
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
NVI0 is up, line protocol is up
  Hardware is NVI
  Interface is unnumbered. Using address of Ethernet0/0 (192.168.5.119)
  MTU 1514 bytes, BW 56 Kbit/sec, DLY 5000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation UNKNOWN, loopback not set
  Keepalive set (10 sec)
  Unknown duplex
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
Router#
Router#
Router#show platform software interface-status

Status of Monitored Interfaces
------------------------------
Linux Interface   IOS  Running  Carrier  MTU    Speed     Duplex   Autoneg
enp1s0f3          1/3  No       No       1500   ???       ???      Yes
enp1s0f2          1/2  No       No       1500   ???       ???      Yes
enp1s0f1          1/1  Yes      Yes      1500   1 Gbps    full     Yes
enp1s0f0          1/0  Yes      Yes      1500   100 Mbps  full     Yes
enp3s0            0/0  Yes      Yes      1500   1 Gbps    full     Yes

Router#
Router#
Router#!!! Captured first iperf3...
Router#

First test results:

C:\Progs\iperf-3.1.3-win64>iperf3.exe -c 192.168.132.52
Connecting to host 192.168.132.52, port 5201
[  4] local 192.168.32.52 port 25203 connected to 192.168.132.52 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   256 KBytes  2.08 Mbits/sec
[  4]   1.01-2.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   2.00-3.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.01-4.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   4.00-5.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   5.01-6.01   sec   128 KBytes  1.05 Mbits/sec
[  4]   6.01-7.00   sec   512 KBytes  4.24 Mbits/sec
[  4]   7.00-8.00   sec   768 KBytes  6.30 Mbits/sec
[  4]   8.00-9.00   sec   768 KBytes  6.28 Mbits/sec
[  4]   9.00-10.00  sec   896 KBytes  7.34 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.25 MBytes  2.73 Mbits/sec                  sender
[  4]   0.00-10.00  sec  3.05 MBytes  2.56 Mbits/sec                  receiver

iperf Done.

I change one of the ports to 10Mbps:

Router#
Router#
Router#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#int e1/0
Router(config-if)#speed 10
Router(config-if)#
Aug 12 15:06:38.074: %LINK-3-UPDOWN: Interface Ethernet1/0, changed state to down
Aug 12 15:06:39.075: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to down
Router(config-if)#
Aug 12 15:06:41.216: %LINK-3-UPDOWN: Interface Ethernet1/0, changed state to up
Aug 12 15:06:42.216: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to up
Router(config-if)#
Router(config-if)#
Router(config-if)#
Router(config-if)#!!! Second iperf3...
Router(config-if)#

And run the second test:

C:\Progs\iperf-3.1.3-win64>iperf3.exe -c 192.168.132.52
Connecting to host 192.168.132.52, port 5201
[  4] local 192.168.32.52 port 25206 connected to 192.168.132.52 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec  1.38 MBytes  11.4 Mbits/sec
[  4]   1.01-2.01   sec  1.12 MBytes  9.47 Mbits/sec
[  4]   2.01-3.01   sec  1.12 MBytes  9.40 Mbits/sec
[  4]   3.01-4.01   sec  1.12 MBytes  9.44 Mbits/sec
[  4]   4.01-5.01   sec  1.12 MBytes  9.42 Mbits/sec
[  4]   5.01-6.01   sec  1.12 MBytes  9.44 Mbits/sec
[  4]   6.01-7.01   sec  1.12 MBytes  9.44 Mbits/sec
[  4]   7.01-8.01   sec  1.12 MBytes  9.43 Mbits/sec
[  4]   8.01-9.01   sec  1.12 MBytes  9.45 Mbits/sec
[  4]   9.01-10.01  sec  1.12 MBytes  9.47 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec  11.5 MBytes  9.64 Mbits/sec                  sender
[  4]   0.00-10.01  sec  11.5 MBytes  9.64 Mbits/sec                  receiver

iperf Done.

Analysis indicates that this problem is caused by network interface offloading.  I use the following script on startup, though it can probably be simplified to just disable gro:

for int in $(ls -1 /sys/class/net/ | egrep "^en"); do
    ethtool -K $int rx off tx off sg off tso off gso off gro off
done

I don't understand why everyone using the 5921 ESR doesn't have this problem.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco