Unable to achieve 10G internet bandwidth on the CSR1000v router

[san ju.]

Hello There,

I have two CSR1000v routers running Cisco IOS XE Software, Version 17.03.04a. All interfaces are operating at 10G speeds, and the throughput level for both routers is set to 10G.

The current throughput level is 10000000 kb/s

Behind the routers, there are two Brocade VDX 6740 switches configured in VCS mode. The servers are connected to these switches via 10G links. The CSR1000v routers are hosted on the servers, and BGP sessions are established with the ISPs from the CSR1000v.

Here are the test results using iperf:

Virtual Machines in the Same VLAN on Different Servers:

- Transfer: 10.5 GBytes Bitrate: 9.00 Gbits/sec

Virtual Machines in Different VLANs on Different Servers:

- Transfer: 3.04 GBytes Bitrate: 2.61 Gbits/sec

Virtual Machine to a Public Server on the Internet:

- Transfer: 3.04 GBytes  Bitrate: 2.60 Gbits/sec

It seems the performance drops significantly when traffic crosses different VLANs or goes to the internet, suggesting something may be off in the CSR1000v. Despite all interfaces and links operating at 10G, I can't identify the root cause of this bottleneck.

Any insights or suggestions to resolve this issue would be greatly appreciated!

Thanks,
Punkn jr

[Richard Burts]

Punkn jr I can understand the drop in performance when going to the Internet. You are going through multiple devices, on multiple links, and there is a very good chance that one (or more) devices or one (or more) links are performing poorly.

I am a bit surprised that vlan to vlan traffic seems to be impacted. Clearly the biggest difference between traffic within the same vlan and traffic between vlans is that we have introduced routing decisions into the forwarding logic. Without knowing more about your configuration it is difficult to know what is causing this impact.

[san ju.]

Hello @Richard Burts,

Thanks for the information, please let me know what specific configuration you would like to see.
FYI - Our public LAN on CSR

GigabitEthernet3 is up, line protocol is up 
  Hardware is CSR vNIC, address is 06da.314d.48b4 (bia 06da.314d.48b4)
  Description: ***PUBLIC_LAN1***
  Internet address is 104.18.10.1/24
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 10000Mbps, link type is force-up, media type is Virtual

ISP interface on CSR:

GigabitEthernet6 is up, line protocol is up 
  Hardware is CSR vNIC, address is 964b.615d.ae66 (bia 964b.615d.ae66)
  Description: COGENT-UPLINK-1
  Internet address is 19.6.110.185/31
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 10000Mbps, link type is force-up, media type is Virtual
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00

BGP config:

router bgp 68731
 bgp log-neighbor-changes
 neighbor 19.6.110.184 remote-as 174
 neighbor 19.6.110.184 local-as 39518
 neighbor 19.6.110.184 description Cogent-1
 !
 address-family ipv4
  network 104.18.10.0 mask 255.255.255.0
  neighbor 19.6.110.184 activate
  neighbor 19.6.110.184 soft-reconfiguration inbound
  neighbor 19.6.110.184 route-map cogent_in in
  neighbor 19.6.110.184 route-map cogent-out out
 exit-address-family
 !

On cogent_in have community no-export and cogent-out just having a prefix list mapped to to without any attribute configuration.

R1#sho ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "bgp 68731", distance 20, metric 0, candidate default path
  Tag 39518, type external
  Last update from 19.6.110.184 1w3d ago
  Routing Descriptor Blocks:
  * 19.6.110.184, from 19.6.110.184, 1w3d ago
      Route metric is 0, traffic share count is 1
      AS Hops 2
      Route tag 39518
      MPLS label: none

#sho ip route
Gateway of last resort is 19.6.110.184 to network 0.0.0.0
B*    0.0.0.0/0 [20/0] via 19.6.110.184, 1w3d

Let me know if you need other configuration.

Thanks,

Punkn jr

[Richard Burts]

Punkn jr Thanks for the outputs that you provided. I am particularly puzzled with the output of show ip route which seems to indicate that it knows the default route but not routes to connected interfaces in your network.

When you ran your test of performance between vlans, can you tell us which vlans those were, and how they are connected (what device(s) does the traffic go through)?

[san ju.]

@Richard Burts - Sorry for that, actually i forget to update the subnet details under the ip route, please find the following,

R1#sho ip route        
Gateway of last resort is 19.6.110.184 to network 0.0.0.0

B*    0.0.0.0/0 [20/0] via 19.6.110.184, 1w4d

      104.0.0.0/8 is variably subnetted, 3 subnets, 3 masks
S        104.18.10.0/23 is directly connected, Null0
C        104.18.10.0/24 is directly connected, GigabitEthernet3
L        104.18.10.1/32 is directly connected, GigabitEthernet3
      19.6.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        19.6.110.184/31 is directly connected, GigabitEthernet6
L        19.6.110.185/32 is directly connected, GigabitEthernet6
      23.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C        23.18.1.0/24 is directly connected, GigabitEthernet5.300
L        23.18.1.1/32 is directly connected, GigabitEthernet5.300

Same VLAN Test:
VM1 (104.18.10.130) in Server 1 on vlan 503 <-LACP-> Switch (VDX 6740) <-LACP-> VM2 (104.18.10.127) in Server 2 on vlan VLAN 503.

Different VLAN Test:
VM1 (104.18.10.130) in Server 1 on vlan 503 <-LACP-> Switch (VDX 6740) <-LACP-> Server 2 CSR1000vR1 VM interface tagged with 503 VIP (104.18.10.3 (GW)) <-LACP-> Switch (VDX 6740) tagged 300 <-LACP-> Server 2 VM on vlan 300 (23.18.1.127).

I hope the information is clear for you, let me know if you need any additional information.

Thanks,

Punkn jr

[Richard Burts]

Punkn jr

Thanks for the output. Am I correct in understanding that the routing decision between subnets is made on CSR1000vR1? Can you post the configuration of this device?

[san ju.]

Hello @Richard Burts @Flavio Miranda @liviu.gheorghe ,

Apologies for the delayed response. To test this again, I deployed a Catalyst 8000v router on a server along with two virtual machines:

VM1: IP - 10.100.100.10/24, Gateway - 10.100.100.1 (Router 1)
VM2: IP - 10.101.101.10/24, Gateway - 10.101.101.1 (Router 1)

The router is a fresh deployment with a throughput of 20G, and no routing configurations have been applied yet. However, when running iperf tests between the two VMs, I’m only getting a maximum throughput of 3 Gbps — the same as the CSR router currently in production.

Do you have any insights into why this might be happening or suggestions for resolving the issue?

Please find the attached cmd result from the c8000vThanks.

[Flavio Miranda]

@san ju. 

where did you get this information of  20G throughput. On the Datasheet I can see this

Table 8.           Minimum server resource requirements per Cisco CSR 1000v instance

Throughput	Technology Package
	IP Base	Security	AppX	AX
10 Mbps	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB
50 Mbps	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB
100 Mbps	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB
250 Mbps	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB
500 Mbps	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB
1 Gbps	1 vCPU/4 GB	1 vCPU/4 GB	1 vCPU/4 GB	2 vCPU/4 GB
2.5 Gbps	1 vCPU/4 GB	2 vCPU/4 GB	4 vCPU/4 GB	4 vCPU/4 GB
5 Gbps	1 vCPU/4 GB	2 vCPU/4 GB	8 vCPU/4 GB	8 vCPU/4 GB
10 Gbps	2 vCPU/4 GB	Not supported	Not supported	Not supported

Cisco Cloud Services Router 1000v Data Sheet - Cisco

[san ju.]

@Flavio Miranda - The requirement say its for CSR1000v, i have tested it on the upgraded version of CSR which is Catalyst 8000v.

Thanks!.

[Flavio Miranda]

Oh sorry. I missed that. We were taking about 1000v before.  Did you check this same information for 8000v ? 

[Flavio Miranda]

@san ju. 

when the router is communicating through different VLANs the traffic is passing through a gateway, right?  The problem could be on the devces in between and not necessarily on the Router.

[san ju.]

@Flavio Miranda - Yes through the Gateway which is the CSR vip(hsrp) - There is not other devices in-between there is only CSR<->switch<->servers. All are opreating at the desired speed.

Thanks,
Punkn jr

[Flavio Miranda]

But thinking about a TAC case, for example, your first scnerio would be used as evidence that the router is working properly. 

[san ju.]

@Flavio Miranda 

In the first scenario, "Virtual Machines in the Same VLAN on Different Servers," I assume the traffic does not reach the router since both VMs are in the same subnet. The switch should handle the frame at layer 2. Please correct me if am wrong.

Thanks,

Punkn jr

[Flavio Miranda]

You are right. I just understood that the traffic was generated from router to router on that scenario.