ISR 4331 Unable to use full bandwidth.

[SMcCoy841]

Greetings all! 

I am having a strange issue with MPLS traffic inbound to the Gig interface of an ISR 4331. The circuit is 100 Mbps. We are having no issues with traffic outbound on that interface. File transfers are quick and traffic will use the full 100 Mbps. The trouble is with traffic inbound to that interface from the MPLS network. File transfers inbound are painfully slow. I did some testing with iPerf and even with 30 parallel streams, the best I can get inbound is approximately 60 Mbps.

I worked through the network path and made sure there were no bottlenecks or errors on my equipment.  I have worked with the telco and they have done all sorts of testing on that circuit.  They claim it will handle the full 100 Mbps in both directions. I worked with TAC and the engineer thinks it could be a hardware limitation. Strange that you would get 100 Mbps in one direction and not the other. The router does have the Performance license so it should have a max throughput of 300 Mbps. We even applied a trial of the Boost license and completely un-throttled it. Still inbound traffic crawls compared to the outbound. 

Not sure what else to look at or try at this point.  I'm hoping someone else has seen this or has some fresh ideas.

sh int gi0/1/0
GigabitEthernet0/1/0 is up, line protocol is up
Hardware is NIM-1GE-CU-SFP, address is 643a.eae2.79e8 (bia 643a.eae2.79e8)
Description: MPLS
Internet address is 12.34.56.78/30
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is auto, media type is RJ45
output flow-control is off, input flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 43019
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
30 second input rate 6000 bits/sec, 5 packets/sec
30 second output rate 9000 bits/sec, 6 packets/sec
17216505 packets input, 18210555876 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
15732187 packets output, 6772967354 bytes, 0 underruns
Output 6 broadcasts (0 IP multicasts)
0 output errors, 5 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

interface GigabitEthernet0/1/0
description MPLS
ip flow monitor Netflow input
ip flow monitor Netflow output
ip address 12.34.56.78 255.255.255.252
load-interval 30
media-type rj45
negotiation auto
no cdp enable
service-policy output 100MBPS_QOS
end

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
sh license feature

[balaji.bandi]

The router does have the Performance license so it should have a max throughput of 300 Mbps. 

you mentioned you have License of 300MB

I do see some drops here :

Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 43019

Can you explain us how is this router connected ? did you check physical cables. have you tested the Link with out Router ?

if possible show run config from router and version ?

[SMcCoy841]

The router connects to the telco equipment using an ethernet cable connected to interface Gig 0/1/0.  I've checked the cable and even replaced it.  Among the other things I have tried is hard coding the interface to 100 Mbps.  The telco didn't recommend that so I switched it back auto-negotiate.  The router is using SMART Licensing but I disabled that temporarily to apply the Boost license.

I've attached the config along with the output of 'show version' and 'show license all' commands.

[balaji.bandi]

Thank you for the information, just to clarify, is this ever worked, or from the start it is limited to the results you posted.

before we get understanding the config, please confirm this test " have you tested the Link with out Router ?" see are you able to get 100MB or near by ?

is this throughput you mentioned 60MB over VPN ?

another point in show output

The current throughput level is unthrottled

but do not see boost license

[SMcCoy841]

This has never worked. The router is located at a remote office and I haven't had the opportunity to try and switch it out.  This is a dedicated MPLS circuit provided by our carrier.  It is not used for internet access.

Thanks!

[balaji.bandi]

Agreed its MPLS, before we suspect what is wrong with Router, we just looking some confirmation, it the MPLS throuput ok with out any router or VPN.

[SMcCoy841]

I have not had a chance to swap out the router or connect the circuit to a different piece of equipment.  Right now I am just assuming that the carrier is correct when they tell me that their testing shows the circuit is capable of 100 Mbps in both directions.

[Leo Laohoo]

Boost License not enabled.

[SMcCoy841]

But the router is showing the current throughput as being unthrottled.

[Leo Laohoo]

Without any license, 4331 can support up to 100 Mbps of aggregate (inbound + outbound) traffic.

That 100 Mbps is not just the WAN speed but every interface, including LAN interface.  

[Joseph W. Doherty]

". . . aggregate (inbound + outbound) traffic."

Are you saying both ingress and egress is counted toward the platform bandwidth aggregation license?

My understanding has been bandwidth is counted as it flows through the router.  I.e. this might be counted as the aggregate sum of all ingress traffic or all egress traffic, but not by counting both.  (Not sure how a flow's traffic is counted, if counted at all, if dropped by an ACL.  I suspect only traffic crossing the backplane, and perhaps router sourced traffic, is counted for bandwidth consumption.)

[Leo Laohoo]

---

@Joseph W. Doherty wrote:

Are you saying both ingress and egress is counted toward the platform bandwidth aggregation license?

---

As far as I am aware, ALL traffic, whether it is WAN or LAN, as long is it hits the router interfaces, are counted towards this bandwidth aggregation.  

[Joseph W. Doherty]

I totally agree that all interfaces' traffic might be counted.  E.g. both WAN and LAN interfaces.

What I'm not certain about, is, if indeed it's interface traffic that's counted, whether all interface traffic, both ingress and egress is counted, or just ingress or egress.

To make the distinction clear, consider a single duplex 100 Mbps copper link.  Being duplex, it can carry 100 Mbps in both directions, concurrently.  So, the 100 Mbps link can have 200 Mbps of traffic.

Now, cut that link, insert a router or switch in-line.  If you count ingress and egress traffic on the two ports, you would have 400 Mbps, yet only 200 Mbps crosses the backplane. So, which does a 4K router count for bandwidth licensing?

[Leo Laohoo]

---

@Joseph W. Doherty wrote:

To make the distinction clear, consider a single duplex 100 Mbps copper link.  Being duplex, it can carry 100 Mbps in both directions, concurrently.  So, the 100 Mbps link can have 200 Mbps of traffic.

---

Correct.  However, the "default" bandwidth supported for a 4331 is 100 Mbps (no Boost License aka "base").  And we all agree this 100 Mbps (base) is "software limited".  

---

@Joseph W. Doherty wrote:

yet only 200 Mbps crosses the backplane. So, which does a 4K router count for bandwidth licensing?

---

(Unless someone from Cisco can correct me, I was made to believe the "backplane" is the where this base license sits.)  It is 100 Mbps whichever direction (inbound &/or outbound).

Hey, I could be wrong (not the first time this has happened) or mis-interpret the data I was provided, but that is what I was told.