Solved: High txload and rxload values for VLAN interface

luis Ibarra · ‎01-22-2016

Hi everyone

I am trying to figure out if some of my VLAN interfaces are having issues. I was looking for packet loss and saw the following on some of my VLANs, see orange highlights below. I am get very high numbers for my tx and rx loads, but my reliability is still 255/255. Is this an issue? I can't find anything online about VLAN interfaces having high tx or rx loads.

Thank you

Luis

Vlan12 is up, line protocol is up
Hardware is Ethernet SVI, address is b07d.4700.1567 (bia b07d.4700.1567)
Internet address is X.X.12.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 255/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 9w1d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1399548000 bits/sec, 131394 packets/sec
5 minute output rate 205000 bits/sec, 122 packets/sec
     731866565450 packets input, 976358742737824 bytes, 0 no buffer
     Received 0 broadcasts (30873 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     904589193 packets output, 458849071216 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

and

Vlan15 is up, line protocol is up
Hardware is Ethernet SVI, address is b07d.4700.1554 (bia b07d.4700.1554)
Internet address is X.X.15.1/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 249/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 9w1d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 979204000 bits/sec, 91685 packets/sec
5 minute output rate 147000 bits/sec, 90 packets/sec
     494602899301 packets input, 661946671057181 bytes, 0 no buffer
     Received 0 broadcasts (19808 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     440301572 packets output, 103441792223 bytes, 0 underruns
     0 output errors, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

Richard Burts · ‎04-20-2016

There is a conceptual thing that we should be careful about as we talk about utilization. It is one thing to look at the utilization of a physical interface such as Gig3/0/12 and quite a different thing to look at the utilization of vlan 12. The original post was asking about utilization of vlan interfaces and then evolved into utilization of physical interfaces. I am not clear whether this most recent post is talking about vlan interfaces or physical interfaces.

When you look at a physical interface there is some bandwidth (which represents capacity of the interface) that is associated with the interface and it is meaningful to compare the actual utilization to the potential max utilization. But the situation is different when you look at the vlan interface, which is a virtual interface. The vlan interface looks like Ethernet so IOS assigns an Ethernet bandwidth to the interface. But in this case the bandwidth of the vlan interface has no relationship to its max capacity. The max capacity of the vlan interface is dependent on the physical interfaces that are in that vlan. Let us think of an example - what if vlan 12 has 12 physical interfaces in the vlan. And let us assume that each of the physical interfaces is receiving at 10% of its capacity - so certainly no utilization problem there. But the vlan interface is now looking like it is 120% of capacity, or rxload is 255/255. It looks like a capacity issue but it is not.

So what interfaces is this new post looking at?

HTH

Rick

HTH

Rick

View solution in original post

Leo Laohoo · ‎01-22-2016

When those rates are that high, what is the output to the command "sh controller util"?

luis Ibarra · ‎01-25-2016

Hi Leo

See below, Is the Receive column actual bandwidth being used, for example port Gi1/0/1, is that 80Mb?

Switch1#sh controllers utilization
Port       Receive Utilization Transmit Utilization
Gi1/0/1      80
Gi1/0/2      110
Gi1/0/3      280
Gi1/0/4      310
Gi1/0/5      120
Gi1/0/6      310
Gi1/0/7      00
Gi1/0/8      150
Gi1/0/9      320
Gi1/0/10      50
Gi1/0/11      30
Gi1/0/12      260
Gi1/0/13      00
Gi1/0/14      00
Gi1/0/15      00
Gi1/0/16      00
Gi1/0/17      00
Gi1/0/18      00
Gi1/0/19      10
Gi1/0/20      00
Gi1/0/21      00
Gi1/0/22      00
Gi1/0/23      00
Gi1/0/24      00
Gi2/0/1      110
Gi2/0/2      230
Gi2/0/3      310
Gi2/0/4      170
Gi2/0/5      270
Gi2/0/6      160
Gi2/0/7      00
Gi2/0/8      280
Gi2/0/9      250
Gi2/0/10      00
Gi2/0/11      00
Gi2/0/12      00
Gi2/0/13      00
Gi2/0/14      00
Gi2/0/15      00
Gi2/0/16      00
Gi2/0/17      00
Gi2/0/18      00
Gi2/0/19      00
Gi2/0/20      00
Gi2/0/21      00
Gi2/0/22      00
Gi2/0/23      00
Gi2/0/24      00
Gi3/0/1      023
Gi3/0/2      00
Gi3/0/3      011
Gi3/0/4      018
Gi3/0/5      00
Gi3/0/6      027
Gi3/0/7      011
Gi3/0/8      00
Gi3/0/9      015
Gi3/0/10      013
Gi3/0/11      00
Gi3/0/12      029
Gi3/0/13      012
Gi3/0/14      00
Gi3/0/15      020
Gi3/0/16      022
Gi3/0/17      00
Gi3/0/18      018
Gi3/0/19      014
Gi3/0/20      00
Gi3/0/21      03
Gi3/0/22      00
Gi3/0/23      00
Gi3/0/24      03
Gi3/0/25      03
Gi3/0/26      01
Gi3/0/27      02
Gi3/0/28      03
Gi3/0/29      05
Gi3/0/30      02
Gi3/0/31      06
Gi3/0/32      05
Gi3/0/33      04
Gi3/0/34      06
Gi3/0/35      02
Gi3/0/36      04
Gi3/0/37      50
Gi3/0/38      10
Gi3/0/39      10
Gi3/0/40      10
Gi3/0/41      20
Gi3/0/42      00
Gi3/0/43      20
Gi3/0/44      00
Gi3/0/45      00
Gi3/0/46      00
Gi3/0/47      10
Gi3/0/48      00
Gi4/0/1      00
Gi4/0/2      027
Gi4/0/3      11
Gi4/0/4      00
Gi4/0/5      028
Gi4/0/6      00
Gi4/0/7      00
Gi4/0/8      021
Gi4/0/9      00
Gi4/0/10      00
Gi4/0/11      026
Gi4/0/12      00
Gi4/0/13      00
Gi4/0/14      020
Gi4/0/15      00
Gi4/0/16      00
Gi4/0/17      015
Gi4/0/18      00
Gi4/0/19      00
Gi4/0/20      012
Gi4/0/21      00
Gi4/0/22      00
Gi4/0/23      00
Gi4/0/24      00
Gi4/0/25      00
Gi4/0/26      01
Gi4/0/27      02
Gi4/0/28      03
Gi4/0/29      04
Gi4/0/30      01
Gi4/0/31      02
Gi4/0/32      03
Gi4/0/33      04
Gi4/0/34      05
Gi4/0/35      00
Gi4/0/36      00
Gi4/0/37      60
Gi4/0/38      40
Gi4/0/39      00
Gi4/0/40      10
Gi4/0/41      00
Gi4/0/42      00
Gi4/0/43      00
Gi4/0/44      00
Gi4/0/45      00
Gi4/0/46      00
Gi4/0/47      00
Gi4/0/48      00

Total Ports : 144
Total Ports Receive Bandwidth Percentage Utilization : 2
Total Ports Transmit Bandwidth Percentage Utilization : 3
Switch 1 Stack Ring Max Percentage Utilization : 1
Switch 2 Stack Ring Max Percentage Utilization : 1
Switch 3 Stack Ring Max Percentage Utilization : 0
Switch 4 Stack Ring Max Percentage Utilization : 0

Leo Laohoo · ‎01-25-2016

See below, Is the Receive column actual bandwidth  being used, for example port Gi1/0/1, is that 80Mb?

No, it is 8% receive and 0% transmit.

Very busy switch. Describe what the model of this stack is and what application is connected? I'm suspecting there are a lot of dropouts on this stack.

luis Ibarra · ‎01-25-2016

Leo

Its a multicast environment, with about 650 IP cameras

All recording is unicast, but all " live viewing" of cameras is multicast. Each camera is a multicast source.

All the recording/storage and viewing stations are in the same vlan and on this stack which happens to be 3850s. The first two switches host all the IDF connections, that is why the output is high on the ports 1 - 10 on them. at any one time there may be about 220 active multicast streams

The config is basic, VLAN routing at the core (this stack) and PIM sparse-dense mode with a single RP hosted at core. All other switch just span the VLAN at the edge

how can we see drops as you mentioned? I have the out put for "show interfaces" and "show interfaces counters errors" all but two ports had issues due to layer one reasons

Leo Laohoo · ‎01-25-2016

Post the output to the following commands:

sh interface Gi1/0/4;
sh interface Gi1/0/9;
sh interface Gi1/0/12;
sh interface Gi3/0/12; and
sh interface Gi4/0/5

What version is the stack running on?

luis Ibarra · ‎01-25-2016

Leo, here you go

I will have to get the version tomorrow, I don't have access at the moment

GigabitEthernet1/0/4 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is bcc4.93f5.e804 (bia bcc4.93f5.e804)
Description: IDF-B
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 80/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:06, output never, output hang never
Last clearing of "show interface" counters 9w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 315648000 bits/sec, 29167 packets/sec
5 minute output rate 35000 bits/sec, 23 packets/sec
     158408364884 packets input, 214138018731464 bytes, 0 no buffer
     Received 17774700502 broadcasts (594815932 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 594815932 multicast, 0 pause input
     0 input packets with dribble condition detected
     165879879 packets output, 68817956884 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

***********************************************************************************

GigabitEthernet1/0/9 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is bcc4.93f5.e809 (bia bcc4.93f5.e809)
Description: Trunks to IDF-D
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 80/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:19, output never, output hang never
--More--           Last clearing of "show interface" counters 9w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 317640000 bits/sec, 29475 packets/sec
5 minute output rate 37000 bits/sec, 25 packets/sec
     154740708967 packets input, 208602915717630 bytes, 0 no buffer
     Received 10122645719 broadcasts (1532639756 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1532639756 multicast, 0 pause input
     0 input packets with dribble condition detected
     137072768 packets output, 29819450088 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

************************************************************************************

GigabitEthernet1/0/12 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is bcc4.93f5.e80c (bia bcc4.93f5.e80c)
Description: Trunk to IDF-2B
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 65/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:17, output never, output hang never
Last clearing of "show interface" counters 9w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 255517000 bits/sec, 25474 packets/sec
--More--           5 minute output rate 51000 bits/sec, 30 packets/sec
     113620728684 packets input, 144844471035811 bytes, 0 no buffer
     Received 34658914601 broadcasts (299162012 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 299162012 multicast, 0 pause input
     0 input packets with dribble condition detected
     279199163 packets output, 200525958912 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

********************************************************************************************

GigabitEthernet3/0/12 is up, line protocol is up (connected)
--More--           Hardware is Gigabit Ethernet, address is f0b2.e5b0.330c (bia f0b2.e5b0.330c)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 74/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 9w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 167364106
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 40000 bits/sec, 28 packets/sec
5 minute output rate 292267000 bits/sec, 27271 packets/sec
     359825658 packets input, 300499771605 bytes, 0 no buffer
     Received 9158334 broadcasts (377350 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 377350 multicast, 0 pause input
     0 input packets with dribble condition detected
     107278590111 packets output, 143985697771295 bytes, 0 underruns
--More--              0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

**********************************************************************************************

GigabitEthernet4/0/5 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is f0b2.e5ed.8705 (bia f0b2.e5ed.8705)
--More--           MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 66/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 9w4d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 4162278147
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 90000 bits/sec, 25 packets/sec
5 minute output rate 261468000 bits/sec, 24673 packets/sec
     192228609 packets input, 169262939813 bytes, 0 no buffer
     Received 199401 broadcasts (196747 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 196747 multicast, 0 pause input
     0 input packets with dribble condition detected
     77342222239 packets output, 103301435224957 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
--More--              0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Leo Laohoo · ‎01-25-2016

Total output drops: 167364106

That one is from Gi 3/0/12.

Total output drops: 4162278147

Next one is from Gi 4/0/15.

Those figures are pretty high.

luis Ibarra · ‎01-26-2016

Hi Leo

Ok I see that, that is were some of my confusion is coming from, I am also looking at this section reliability 255/255, txload 66/255, rxload 1/255.

I will look into the ports and the devises it supports, I believe they are recorders.

thanks

Luis

wongpye01 · ‎04-20-2016

Hi Luis,

I have the same issue on my 6506-E, but do you have any problem with that?

Or did you solve the problem? Please share. Thanks.

Richard Burts · ‎04-20-2016

There is a conceptual thing that we should be careful about as we talk about utilization. It is one thing to look at the utilization of a physical interface such as Gig3/0/12 and quite a different thing to look at the utilization of vlan 12. The original post was asking about utilization of vlan interfaces and then evolved into utilization of physical interfaces. I am not clear whether this most recent post is talking about vlan interfaces or physical interfaces.

When you look at a physical interface there is some bandwidth (which represents capacity of the interface) that is associated with the interface and it is meaningful to compare the actual utilization to the potential max utilization. But the situation is different when you look at the vlan interface, which is a virtual interface. The vlan interface looks like Ethernet so IOS assigns an Ethernet bandwidth to the interface. But in this case the bandwidth of the vlan interface has no relationship to its max capacity. The max capacity of the vlan interface is dependent on the physical interfaces that are in that vlan. Let us think of an example - what if vlan 12 has 12 physical interfaces in the vlan. And let us assume that each of the physical interfaces is receiving at 10% of its capacity - so certainly no utilization problem there. But the vlan interface is now looking like it is 120% of capacity, or rxload is 255/255. It looks like a capacity issue but it is not.

So what interfaces is this new post looking at?

HTH

Rick

HTH

Rick

wongpye01 · ‎04-24-2016

Thanks Richard,

I have the same issue on the int vlan.

So what you mean is if the phy port utilization is ok, and only int vlan have this problem, should be ok? Am i right?

Thanks!

Richard Burts · ‎04-25-2016

Yes you are right.

HTH

Rick

HTH

Rick

luis Ibarra · ‎04-26-2016

Richard

Thank you for your explanation. The "high" utilization on our VLAN interfaces drove us nuts.

The issue on our physical ports is being caused by how a video recorder manager controls its storage box members. As recorded video is played back, all request go to the manager and it causes the high output drops.

again, thank you

Luis

Richard Burts · ‎04-26-2016

Luis

You are quite welcome. I am glad that my explanation was helpful. Thank you for the update about your situation and what was causing the drops.

HTH

Rick

HTH

Rick