How to troubleshoot something like Rxload compare to bandwitdh?

Ken Lee · ‎04-17-2011

I have this issue below which the RXload is alway high above 200/255 but I did a sh ip accounting, Not sure how i can prove it to the apps side that these are the traffic load. Would need more advise on expert whether is it the configuration or really the load traffic problem.

Thanks!

----------------

sh int gi0/0:
--------------
GigabitEthernet0/0 is up, line protocol is up
Hardware is MV96340 Ethernet, address is 001d.7015.9f40 (bia 001d.7015.9f40)
Internet address is 126.110.254.254/30
MTU 1500 bytes, BW 20000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 67/255, rxload 209/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is T
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2d16h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 6287
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
5 minute input rate 17640000 bits/sec, 2571 packets/sec
5 minute output rate 5325000 bits/sec, 1860 packets/sec
     92413970 packets input, 1786666425 bytes, 0 no buffer
     Received 56544 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     91313207 packets output, 2844484686 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

class-map match-any class1-video
match dscp cs4
match dscp af41
match dscp af42
match dscp af43
class-map match-any class1-voice
match dscp ef
class-map match-any class-Network-Critical
match access-group name acl-Routing-Protocol
class-map match-any class4
match dscp cs1
match dscp af11
match dscp af12
match dscp af13
class-map match-any class2
match dscp af31
match dscp af32
match dscp af33
class-map match-any class2-signalling
match dscp cs3
!
policy-map CBWFQ_20M
class class-Network-Critical
    bandwidth remaining percent 5
class class1-voice
    priority percent 10
class class1-video
    bandwidth remaining percent 5
class class2-signalling
    bandwidth remaining percent 5
class class2
    bandwidth remaining percent 25
class class4
    bandwidth remaining percent 9
class class-default
    bandwidth remaining percent 50
policy-map shape_20M
class class-default
    shape average 20000000
service-policy CBWFQ_20M
!
!
!
!
interface Loopback0
ip address 126.110.255.249 255.255.255.255
!
interface GigabitEthernet0/0
bandwidth 20000
ip address 126.110.254.254 255.255.255.252
no ip redirects
no ip proxy-arp
ip accounting output-packets
ip accounting access-violations
rate-limit input access-group 105 7000000 1000000 2000000 conform-action transmit exceed-action drop
rate-limit output access-group 105 7000000 1000000 2000000 conform-action transmit exceed-action drop
duplex full
speed 100
service-policy output shape_20M
!
interface GigabitEthernet0/1
ip address 126.110.252.6 255.255.255.252
no ip redirects
no ip proxy-arp
ip accounting access-violations
duplex full
speed 1000
!
router eigrp 8
redistribute rip metric 15000 500 255 1 1500
passive-interface default
no passive-interface GigabitEthernet0/0
no passive-interface GigabitEthernet0/1
network 126.110.0.0 0.7.255.255
no auto-summary
no eigrp log-neighbor-changes
!
ip forward-protocol nd
no ip forward-protocol udp tftp
no ip forward-protocol udp nameserver
no ip forward-protocol udp domain
no ip forward-protocol udp time
no ip forward-protocol udp netbios-ns
no ip forward-protocol udp netbios-dgm
no ip forward-protocol udp tacacs
!
no ip http server
no ip http secure-server

..

...

....

.....

Leo Laohoo · ‎04-17-2011

1. What is the Uptime of this machine?

2. You have a Gig interface and you're running at FastEthernet?

3. Total output drops: 6287? In "2d16h"? That's way too high.

I think you're interface is dropping because you're pushing too much than your FE can handle.

samavedula_rama · ‎04-18-2011

@leolaohoo

I believe Ken is talking about the RXLoad being very high. So, that has nothing to do with the output drops, does it?

@Ken

Well, I believe you have only a output service policy configured. And yes, if you have some sort of input policy configured too that would match the most common traffic that your apps people use? May be a monitoring software can help. What do you use for your network?

And I see you have this

rate-limit input access-group 105 7000000 1000000 2000000 conform-action transmit exceed-action drop

Do you have the log keyword for the access-list set? May be you should check how much traffic that access-list is pushing towards you. I don't see an issue with your output policy or anything else for that matter.

Ken Lee · ‎04-18-2011

Anyway to check the input policy is catered enough for the bandwitdh? No monitoring software on the network.

And I see you have this

I can provide the logs for the access drop for the rate limit if you need to see the logs.

Thanks,

Regards,

KA

Ken Lee · ‎04-18-2011

Leo,

Thanks for your reply.

Well, on the router, I have checked the log,

Apr 11 01:25:48.249 GMT: %MV64340_ETHERNET-5-LATECOLLISION: GigabitEthernet0/0,
late collision error

The GiG Ethernet was set to Speed 100M because it is a Metro-E connection. Probably I should put to speed 1000M instead?

Yes, there is a total output of drops which is still increasing.. I check the queue of the interface is 64packets queueing.

Thanks,
KA

samavedula_rama · ‎04-18-2011

Apr 11 01:25:48.249 GMT: %MV64340_ETHERNET-5-LATECOLLISION: GigabitEthernet0/0,
late collision error

---> This shouldn't matter if you have one or 2 over a longer period of time.

The GiG Ethernet was set to Speed 100M because it is a Metro-E connection. Probably I should put to speed 1000M instead?

--> Depends.How is your service provider set up? Does the port sending you traffic has any QoS configured? I don't see any, since you don't have any inbound policy configured. Generally if there is DSCP configured, the QoS policy must be configured on a per-hop basis. So, worth trying.

Thanks.

Leo Laohoo · ‎04-18-2011

I believe Ken is talking about the RXLoad being very high. So, that has nothing to do with the output drops, does it?

It's related. Why are you having output drops? Why is your RXLoad high? It's all related.

Ken,

Do you have any policy shaping? I'm suspecting your router is pushing as hard as it can but FastEthernet speed can only push a fraction of the amount and droping. The RXLoad could be high due to retries.

Ken Lee · ‎04-18-2011

Leo,

You are right. Pushing too hard. I have QoS as per the config but I am thinking I can prove to the application side that it is really the traffic that jamming the bandwitdh. How can I find those valuable info from the router side?

Thanks,

KA