strange port utilization

Mk. Jone · ‎02-10-2016

Hi,

I have a very high utilization on one of my 2960 port! this port is connected to my hp server. I got this server when my server is not booting with os.

show interface gi1/0/15

GigabitEthernet1/0/15 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10bd.1839.1a0f (bia 10bd.1839.1a0f)
Description:
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 193/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec
     0 packets input, 810381190528 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, -222695775 pause input
     0 input packets with dribble condition detected
     38854 packets output, 4601437 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Here is my config:

Cisco IOS : Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)

interface GigabitEthernet1/0/15
switchport access vlan 71
switchport mode access
spanning-tree portfast
!

any help?!

Mark Malone · ‎02-11-2016

So the server is receiving a high load is that the issue ?

If your trying to figure out what traffic it is or why its like this first need to identify the traffic , setup a local span session on the port and capture the traffic see exactly what type of traffic it is that should help identify what's wrong . nothing wrong with your config or port otherwise looks fine

monitor session 1 source interface g1/0/15

monitor session 1 destination interface g1/0/1

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/10570-41.html

EDIT: Just to add as well these switches are not built for servers there built pc/phones/prinetrs low end access switches the buffers on the ports are not able to handle large amounts of traffic from servers like distribution switches

pwwiddicombe · ‎02-11-2016

Can you do a "clear counters int g1/0/15" and look again? Really odd that you have

0 packets input, 810381190528 bytes, 0 no buffer. Do any other "active" ports show this indication as well?

Mk. Jone · ‎02-12-2016

hi,

I have another port just like this, but other ports work without any abnormal utilization. its connected to the same server.

#show int gi1/0/16
GigabitEthernet1/0/16 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10bd.1839.1a10 (bia 10bd.1839.1a10)
Description:
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 194/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 6
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 9796000 bits/sec, 17256 packets/sec
     0 packets input, 23211473306688 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 1902017563 pause input
     0 input packets with dribble condition detected
     1553551114 packets output, 201367275195 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

clear counters g1/0/15
Clear "show interface" counters on this interface [confirm]

#show int gi1/0/15
GigabitEthernet1/0/15 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 10bd.1839.1a0f (bia 10bd.1839.1a0f)
Description:
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 194/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:00:02
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec
     0 packets input, 192704128 bytes, 0 no buffer
     Received 0 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 3011002 pause input
     0 input packets with dribble condition detected
     3 packets output, 293 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Mark Malone · ‎02-14-2016

That's strange you see nothing in the pcap ,out of interest if you remove server from that port and attach a standard laptop do you see the same behaviour , it could be the server causing it if not might be an ios issue you could try upgrade to a newer MD release

Mk. Jone · ‎02-22-2016

I found out the reason!

show interface GigabitEthernet1/0/15 flowcontrol
Port       Send FlowControl Receive FlowControl RxPause TxPause
           admin    oper     admin    oper
--------- -------- -------- -------- --------    ------- -------
Gi1/0/15   Unsupp. Unsupp. off      off         1290281413704 0

why I have so many pause frame?!

how can I disable it?!

I read this:

https://supportforums.cisco.com/discussion/10534891/mac-pause-and-flooing

It says that i should disable port fast .

pwwiddicombe · ‎02-23-2016

The pause frames are basically saying the input can't be handled "at the moment" and for the transmitter to stop sending momentarily. If you have SOME active communications, then settings like this can be adjusted; but it looks like the sender (server) is doing something the switch doesn'r really recognize at all.

At this point, I would STRONGLY suggest you attach a working laptop to the port, just to verify the switchport is OK; but would suggest the issue is with either cable, NIC, or configuration on the server. The fact that (a lot of) input is seen but NO packets implies no valid frame has been seen.

Mk. Jone · ‎02-28-2016

As you said I connect pc to bad port . it was OK.

I remove portfast settings from my port and connect my server again. after 4days it seems that the problem has been solved!

TNX

pwwiddicombe · ‎02-17-2016

I see both ports are locked at 1 gig. Are you sure the server is set the same way?

That many bytes and no recognizable input packets is really strange. Compare the configuration of the server and the switch ports to make sure there isn't something mismatched. What kind of server is it? Anything unusual in the network definition or framing?

Mk. Jone · ‎02-17-2016

both port are connected to hp dl380 server. my switch farm contains one 3750 as core switch and many 2960 as dist switch. it was working good but since 3 month ago the problem started!

pwwiddicombe · ‎02-18-2016

As another post recommended, did you actually try to connect a "normal" PC or another server to these ports? (just to be sure it isn't the physical port). If/when you do this, you might have to change something on the port config, if you had set up trunking or something "unique" for the server.

Also, what is the config on the ports in question; both on the Cisco side (and I don't know if it's possible to get the equivalent from the DL380). Does the 380 perhaps have jumbo frames configured, which the Cisco isn't set up for?

Mk. Jone · ‎02-12-2016

I run tcpdump on linux box connected to the port. no packet!!!

As you see I have no input packet! in interface status. It's really strange!!!

5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec

Mk. Jone · ‎04-22-2016

Oh, After 15 days the problem raised again!

Still have the high traffic of pause frames

I Changed :

the cable, server, switch port

getting confused!

pwwiddicombe · ‎04-23-2016

Do you have multiple ports connected to the same server? From reading through again, it looks like you do, and it has issues as well. Note that an ILO connection doesn't count as a data link, so it's not counted and shouldn't cause any issues.

How/why are multiple ports connected, and how is it set up on the server? Is it a connection to multiple vlans; or for backup purposes, or for throughput?

Any of these require careful consideration. If you have them set up for port bundling/etherchannel, the settings between server and switch have to match. Given the type of switch, you might be better off NOT having multiple links to the switch.

If on different subnets, you probably DON'T want this server to be routing.

Is this server set up as VMWare, or a simple/single server? When it gets unresponsive/loading, what do the network statistics on the server show?

Mk. Jone · ‎04-25-2016

I have a Linux cluster with 2node!

my servers have the same hw/sw configuration. all connected to the same switch.

All of my server were working for 2 years without any problem but since 2-3 month ago one of them have a problem. flooding phase frame on NIC port.

my server has 3 connected port !(1 ilo , 2 NICs ). ilo port is OK but 2NICs have problem!

one of my NICs is in access port and the other port is trunk.

I don't know if running Linux command to turn off flow control helps or not.