ISR 4331 Unable to use full bandwidth.

[SMcCoy841]

Greetings all! 

I am having a strange issue with MPLS traffic inbound to the Gig interface of an ISR 4331. The circuit is 100 Mbps. We are having no issues with traffic outbound on that interface. File transfers are quick and traffic will use the full 100 Mbps. The trouble is with traffic inbound to that interface from the MPLS network. File transfers inbound are painfully slow. I did some testing with iPerf and even with 30 parallel streams, the best I can get inbound is approximately 60 Mbps.

I worked through the network path and made sure there were no bottlenecks or errors on my equipment.  I have worked with the telco and they have done all sorts of testing on that circuit.  They claim it will handle the full 100 Mbps in both directions. I worked with TAC and the engineer thinks it could be a hardware limitation. Strange that you would get 100 Mbps in one direction and not the other. The router does have the Performance license so it should have a max throughput of 300 Mbps. We even applied a trial of the Boost license and completely un-throttled it. Still inbound traffic crawls compared to the outbound. 

Not sure what else to look at or try at this point.  I'm hoping someone else has seen this or has some fresh ideas.

sh int gi0/1/0
GigabitEthernet0/1/0 is up, line protocol is up
Hardware is NIM-1GE-CU-SFP, address is 643a.eae2.79e8 (bia 643a.eae2.79e8)
Description: MPLS
Internet address is 12.34.56.78/30
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is auto, media type is RJ45
output flow-control is off, input flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 43019
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
30 second input rate 6000 bits/sec, 5 packets/sec
30 second output rate 9000 bits/sec, 6 packets/sec
17216505 packets input, 18210555876 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
15732187 packets output, 6772967354 bytes, 0 underruns
Output 6 broadcasts (0 IP multicasts)
0 output errors, 5 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

interface GigabitEthernet0/1/0
description MPLS
ip flow monitor Netflow input
ip flow monitor Netflow output
ip address 12.34.56.78 255.255.255.252
load-interval 30
media-type rj45
negotiation auto
no cdp enable
service-policy output 100MBPS_QOS
end

[Joseph W. Doherty]

Leo, unsure what you believe is counted.

If you believe, as I, that the bandwidth is counted like my example of a duplex link, i.e. flows crossing the backplane and not the sum of all ports ingress and egress, then we're in agreement.  (At least I believe that's, more-or-less, how the bandwidth licenses count bandwidth toward the license limits.  [But just as you also note, it wouldn't be the first time I was mistaken {just like the time I thought I was mistaken, but wasn't - laugh}.])

In other words, if a ISR 4K router had a 200 Mbps bandwidth cap, it would support two 100 Mbps ports, it would support them, full rate.  If, however, both ports combined ingress and egress bandwidths were counted, you would need a 400 Mbps license.

In either case, regardless of the number of ports, either method would apply to all of the router's ports.

[SMcCoy841]

The router is currenlty running on an Boost evaluation license.  I will set the throughput back to 300000 and enable Smart Licensing.  During the off hours I will reboot the router to see if that makes a difference.

[Leo Laohoo]

---

@SMcCoy841 wrote:

The current throughput level is unthrottled

---

Wait a second, I know what this is:  CSCvy45932

[chrihussey]

You connect to the MPLS provider on a Gig interface that negotiates to gig and I assume they rate limit you to 100Mbps. I would check with the provider to see if their policies are correct and possibly as a test temporarily remove them to see if there is any improvement in performance.

[SMcCoy841]

I have had two different engineers from our carrier check the circuit and both came back saying their policies are markings are correct.  They do not see any dropped packets on their side.

Thanks!

[Joseph W. Doherty]

In your bandwidth testing, you mention using iPerf - I don't recall whether iPerf offers it, but did any of your testing use UDP?  (Reason I ask, TCP has its own issues that can cause throughput limitation making it more difficult to identify media issues.)  If not, try a test send a tad more than 100 Mbps and see what rate you receive.

BTW, generally, at least in my experience, service providers seldom find unusual causes of performance issues in they're network, unless it's real obvious, without you showing them the specific part of their network having an issue.  On the other hand, also usually, their networks are not the cause issue of many performance issues.

I see you use a service policy, with a shaper, to limit your MPLS egress to 100 Mbps.  Assuming your WAN link is also contracted for 100 Mbps, how to you manage not overrunning the contracted rate for ingress?

[SMcCoy841]

Totally agree with you about the service providers.  That's why I opened two support tickets.  I wanted to be sure they were doing more than just running basic tests or just saying everything looked OK.  More than once I have logged a support ticket for a bouncing circuit and had the support engineer tell me there were no problems while the issue magically fixed itself.

For the ingress traffic, I will look into creating an inbound QoS policy.  I guess the 'Drinking from a fire hose' school of thought would not work very well here.  

[Joseph W. Doherty]

Drinking from the "fire hose", I believe, shouldn't apply in this case to the router, itself, unless your are indeed hitting CPU or bandwidth caps on the router.

What I thinking about is hitting bandwidth limits within the MPLS network.  Your egress shaper should avoid that, but what about ingress traffic?  I.e. how do you know you're not exceeding 100 Mbps, in the MPLS network, to your router?