Router 2921 + EHWIC-D8ESG + EHWIC-4ESG = slow switching performance

[EnricoSavazzi]

As the title says, I have a 2921 with one EHWIC-D8ESG and one EHWIC-4ESG (plus one EHWIC-1GE-SFP-CU, which likely does not contribute to the problem). Both Gigabit multi-port EHWICs are connected to VLAN 10, configured as NAT inside. G0/1, G0/2 and G0/0/0 are all configured as NAT inside. G0/0 is NAT outside, both static and dynamic. So in total I have four separate IP networks working as NAT inside toward a fifth (NAT outside) IP network.

The switching, NAT, routing and overall connectivity all work as intended. However, IP traffic between PCs connected to ports of the EHWICs (including ports on the same EHWIC) is relatively slow and tops out at about 5 MB/second, even when the other router interfaces are doing nothing. With the same PCs connected to a small current-generation 1 Gbit smart switch (D-Link DGS-1210), I can easily exceed 30 MB/second in spite of multiple asymmetric VLANS, trunk ports, aggregated links and ACLs being configured on the switch.

Is there anything I need to configure on the 2921 to get faster local traffic between EHWIC Gbit ports? Or is this level of performance all that the EHWICs + 2921 can provide? If the latter is true, then I probably need to use a modern stand-alone switch instead of the EHWICs.

The current 2921 configuration:

C2921#sh run
Building configuration...

Current configuration : 6951 bytes
!
! Last configuration change at 14:25:15 UTC Fri Feb 26 2021 by ######
! NVRAM config last updated at 14:25:23 UTC Fri Feb 26 2021 by ######
! NVRAM config last updated at 14:25:23 UTC Fri Feb 26 2021 by ######
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname C2921
!
boot-start-marker
boot-end-marker
!
!
enable secret 4 #######################
!
no aaa new-model
!
no ipv6 cef
ip source-route
ip cef
!
!
!
ip dhcp excluded-address 192.168.2.1 192.168.2.2
ip dhcp excluded-address 192.168.2.6 192.168.2.254
ip dhcp excluded-address 192.168.1.1 192.168.1.6
ip dhcp excluded-address 192.168.1.10 192.168.1.254
ip dhcp excluded-address 192.168.1.1 192.168.1.2
ip dhcp excluded-address 192.168.1.7 192.168.1.254
ip dhcp excluded-address 192.168.2.7 192.168.2.254
ip dhcp excluded-address 192.168.3.1 192.168.3.2
ip dhcp excluded-address 192.168.3.7 192.168.3.254
ip dhcp excluded-address 192.168.10.1 192.168.10.2
ip dhcp excluded-address 192.168.10.7 192.168.10.254
ip dhcp excluded-address 192.168.10.1 192.168.10.14
ip dhcp excluded-address 192.168.10.23 192.168.10.254
ip dhcp excluded-address 192.168.2.1 192.168.2.6
ip dhcp excluded-address 192.168.2.11 192.168.2.254
ip dhcp excluded-address 192.168.3.1 192.168.3.10
ip dhcp excluded-address 192.168.3.15 192.168.3.254
!
ip dhcp pool GUEST2
 network 192.168.2.0 255.255.255.0
 domain-name GUEST2.local
 default-router 192.168.2.1
 dns-server 192.168.8.1 8.8.8.8 8.8.4.4
!
ip dhcp pool GUEST1
 network 192.168.1.0 255.255.255.0
 domain-name GUEST1.local
 default-router 192.168.1.1
 dns-server 192.168.8.1 8.8.8.8 8.8.4.4
!
ip dhcp pool GUEST3
 network 192.168.3.0 255.255.255.0
 domain-name GUEST3.local
 default-router 192.168.3.1
 dns-server 192.168.8.1 8.8.8.8 8.8.4.4
!
ip dhcp pool SWITCH
 network 192.168.10.0 255.255.255.0
 domain-name SWITCH.local
 default-router 192.168.10.1
 dns-server 192.168.8.1 8.8.8.8 8.8.4.4
!
!
ip domain name ######.LOCAL
multilink bundle-name authenticated
!
!
crypto pki token default removal timeout 0
!
!
license udi pid CISCO2921/K9 sn FCZ160270EV
!
!
vtp mode transparent
username ####### password 0 ############
!
!
vlan 10
!
ip ssh version 2
!
!
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
interface GigabitEthernet0/0
 description ##HOME##
 ip address 192.168.8.2 255.255.255.0
 ip nat outside
 ip virtual-reassembly in
 ip tcp adjust-mss 1452
 duplex full
 speed 1000
!
interface GigabitEthernet0/1
 description ##GUEST1##
 ip address 192.168.1.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ip tcp adjust-mss 1452
 media-type rj45 auto-failover
!
interface GigabitEthernet0/2
 description ##GUEST2##
 ip address 192.168.2.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ip tcp adjust-mss 1452
 duplex full
 speed 1000
!
interface GigabitEthernet0/0/0
 description ##GUEST3##
 ip address 192.168.3.2 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ip tcp adjust-mss 1452
 media-type rj45 auto-failover
!
interface GigabitEthernet0/1/0
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/1/1
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/1/2
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/1/3
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/0
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/1
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/2
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/3
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/4
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/5
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/6
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface GigabitEthernet0/3/7
 switchport access vlan 10
 no ip address
 duplex full
 storm-control multicast level 70.00 30.00
!
interface Vlan1
 no ip address
!
interface Vlan10
 description ##SWITCH##
 ip address 192.168.10.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip nat inside source list 1 interface GigabitEthernet0/0 overload
ip nat inside source static 192.168.1.3 192.168.8.3
ip nat inside source static 192.168.1.4 192.168.8.4
ip nat inside source static 192.168.1.5 192.168.8.5
ip nat inside source static 192.168.1.6 192.168.8.6
ip nat inside source static 192.168.2.7 192.168.8.7
ip nat inside source static 192.168.2.8 192.168.8.8
ip nat inside source static 192.168.2.9 192.168.8.9
ip nat inside source static 192.168.2.10 192.168.8.10
ip nat inside source static 192.168.3.11 192.168.8.11
ip nat inside source static 192.168.3.12 192.168.8.12
ip nat inside source static 192.168.3.13 192.168.8.13
ip nat inside source static 192.168.3.14 192.168.8.14
ip nat inside source static 192.168.10.15 192.168.8.15
ip nat inside source static 192.168.10.16 192.168.8.16
ip nat inside source static 192.168.10.17 192.168.8.17
ip nat inside source static 192.168.10.18 192.168.8.18
ip nat inside source static 192.168.10.19 192.168.8.19
ip nat inside source static 192.168.10.20 192.168.8.20
ip nat inside source static 192.168.10.21 192.168.8.21
ip nat inside source static 192.168.10.22 192.168.8.22
ip route 0.0.0.0 0.0.0.0 192.168.8.1
!
access-list 1 permit 192.168.1.0 0.0.0.255
access-list 1 permit 192.168.2.0 0.0.0.255
access-list 1 permit 192.168.3.0 0.0.0.255
access-list 1 permit 192.168.10.0 0.0.0.255
!
!
!
control-plane
!
!
banner login ^C
------------------------------------------------------------------
Only for authorized personnel.
Violators will be prosecuted to the full extent of the law.

All operation is logged.
------------------------------------------------------------------
^C
!
line con 0
 session-timeout 30
 exec-timeout 30 0
 logging synchronous
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport input all
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 session-timeout 30
 exec-timeout 30 0
 login local
 transport input ssh
!
scheduler allocate 20000 1000
end

[Reza Sharifi · Accepted solution]

Hi,

To get a faster speed, you probably need a more modern, stand-alone switch, as the 2900 series routers are at least 10 years old and have been EOL/EOS for a long time now. This topic also has been discussed before here a few times. See link for one example:

https://community.cisco.com/t5/routing/cisco-isr-2900-throughput/td-p/3908031

HTH

[Joseph W. Doherty · Accepted solution]

Your bottleneck is likely the performance capacity of your 2921, which is not really suitable for gig LAN routing.  You might further confirm this by checking the 2921's CPU performance history.

Like Reza, I too think, for your interior LAN, you likely would benefit from a stand alone switch.  However, to be clear, if your routing between subnets within you LAN, you'll want a switch with some L3 routing capability.  (NB: some of the newer Cisco "L2" switches support very basic routing.)

Restrict your 2921 to only processing LAN<to/from>WAN traffic.

You also asked is there any config change you could make to improve performance.  Possibly.  The goal is don't do anything you don't need to do, and what you need to do, do it as efficiently as possible.

For example of efficiently, for ACL list 1, sequence the ACEs by hit frequency and/or combine ACEs when possible.  An example of combined ACEs, I believe:

access-list 1 permit 192.168.2.0 0.0.0.255
access-list 1 permit 192.168.3.0 0.0.0.255

might be replaced by:

access-list 1 permit 192.168.2.0 0.0.1.255

Or if your four /24 networks were all out of the same /22, one ACE could be used.

I also notice you're using:

ip tcp adjust-mss 1452

PPPoE?  If so, it's really only needed on the WAN interface.

[EnricoSavazzi]

Thanks, I implemented the configuration changes you suggested. I replaced all ACEs with access-list 1 permit 192.168.0.0 0.0.255.255, and set no ip tcp adjust-mss 1452 on NAT inside interfaces. The mobile gateway does not use PPPoE as far as I know, but loses contact with the router if the router does not use TCP adjust-mss on the WAN interface.

I repeated the speed tests at night time (to avoid most network traffic not generated by my test machines) and got better results, with uncompressed PC-to-PC FTP transfers up to about 11 MB/sec. I cannot be sure how much the configuration changes did help.

Router CPU utilization runs at 10% or less during the heaviest FTP transfers between EHWIC ports, so this is not the limiting factor. It got up to 60% for a few seconds while also doing a copy run start. CPU reaches 20% while downloading/uploading from/to the Internet as fast as my 4G mobile gateway allows (90 Mbit/s down, 2.8 Mbit/s up). The higher CPU utilization in this case is likely due to NAT in the router, which is a must in my configuration.

It may be good to know for the many owners of these old routers that they can still perform well where high throughput is not needed. Using the multi-port Gbit Ethernet EHWICs, on the other hand, may not be a good idea.

[Joseph W. Doherty]

BTW, I forgot to mention, an EHWIC module is limited to 800 Mbps.