enable flow-control on ASR9K interface

Eric Guo · ‎02-14-2014

Hi Group,

Could someone get me a clear understanding about flow-control feature enable on our ASR 9K device ? I tried to enable this feature under my interface, and commit "flow-control ingress/egress/bidirectional" command and "negotiation auto" command in placed already, and I reset the interface by shutdown/no shutdown. But flow-control is still showing off.

Does it mean flow-control is only working for layer 2 interface type ?

Detail information below.

Thanks very much in advance if you can share with your idea !

our end confguration ASK9010 version 3.9.2

interface GigabitEthernet0/2/0/23

flow-control ingress

negotiation auto

transceiver permit pid all

!

interface GigabitEthernet0/2/0/23.429

ipv4 address 10.10.10.1 255.255.255.252

flow ipv4 monitor fmm1 sampler fsm1 ingress

encapsulation dot1q 429

!

peering end configuration ASK1002 XE OS

interface GigabitEthernet0/2/0

no ip address

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

logging event link-status

logging event subif-link-status

load-interval 30

negotiation auto

ntp disable

no mop enabled

!

interface GigabitEthernet0/2/0.429

encapsulation dot1Q 429

ip address 10.10.10.2 255.255.255.252

ip access-group 115 in

no ip redirects

no ip unreachables

no ip proxy-arp

ntp disable

------ interface output from our end after reset interface

igabitEthernet0/2/0/23 is up, line protocol is up

Interface state transitions: 25

Hardware is GigabitEthernet, address is f866.f214.362f (bia f866.f214.362f)

Internet address is Unknown

MTU 1514 bytes, BW 1000000 Kbit

reliability 255/255, txload 3/255, rxload 3/255

Encapsulation ARPA,

Full-duplex, 1000Mb/s, LXFDX, link type is autonegotiation

output flow control is off, input flow control is off

loopback not set,

ARP type ARPA, ARP timeout 04:00:00

Last input 00:00:00, output 00:00:00

Last clearing of "show interface" counters 2d00h

5 minute input rate 15512000 bits/sec, 7098 packets/sec

5 minute output rate 15529000 bits/sec, 2132 packets/sec

2309688790 packets input, 542130815304 bytes, 30871 total input drops

30870 drops for unrecognized upper-level protocol

Received 0 broadcast packets, 117943 multicast packets

0 runts, 0 giants, 0 throttles, 0 parity

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

3337749329 packets output, 4368379414104 bytes, 261493 total output drops

Output 76 broadcast packets, 0 multicast packets

0 output errors, 0 underruns, 0 applique, 0 resets

0 output buffer failures, 0 output buffers swapped out

4 carrier transitions

GigabitEthernet0/2/0/23.429 is up, line protocol is up

Interface state transitions: 13

Hardware is VLAN sub-interface(s), address is f866.f214.362f

Internet address is 66.97.23.185/30

MTU 1518 bytes, BW 1000000 Kbit

reliability 246/255, txload 3/255, rxload 3/255

Encapsulation 802.1Q Virtual LAN, VLAN Id 429, loopback not set,

ARP type ARPA, ARP timeout 04:00:00

Last input 00:00:00, output 00:00:00

Last clearing of "show interface" counters 2d00h

5 minute input rate 15522000 bits/sec, 7131 packets/sec

5 minute output rate 15487000 bits/sec, 2141 packets/sec

2309573924 packets input, 542103669918 bytes, 87167 total input drops

87069 drops for unrecognized upper-level protocol

Received 0 broadcast packets, 87069 multicast packets

3337893541 packets output, 4368493331355 bytes, 16 total output drops

Output 0 broadcast packets, 0 multicast packets

xthuijs · ‎02-16-2014

Flow control at the ethernet level is negotiated by the two end points.

It looks like your peer may not want to participate in this and therefore it is showing off.

You can verify what was negotiated with the show controller gig all command to see what happened down

at that autoneg level.

regards

xander

Eric Guo · ‎02-17-2014

Hi Alex,

Below is short output under my interface.and detail show controller gi 0/2/0/23 all in attachment. please check.

And also under my lab testing with two ASR 9010 routers, the flow-control feature could be able to enable both sides, but only when I disable "negotiation auto" under the interfaces. So back to in this case, my peering device is XE OS device, in these document, there is not explicit command to enable flowcontrol feature under the interface level. there is only saying that the flowcontrol could be able to enable by negotiation auto feature enable. if is this confiict here ? as ASR 9K enable flow-control only when disable " negotiation auto" and XE 1000K router is only can be enable flow-control feature when it learnt from peering with "negotiation auto" is on.

so is this ture ? did I mis-understand here ? and what's negotiation auto feature can do ?

thanks, could we know ?

========================

sh controllers gi 0/2/0/23

Mon Feb 17 14:43:33.997 EST

Operational data for interface GigabitEthernet0/2/0/23:

State:

Administrative state: enabled

Operational state: Up

LED state: Green On

Media:

Media type: X fiber over long-wl laser PMD, full duplex

Optics:

Vendor: CISCO-SUMITOMO

Part number: SCP6G44-C1-BMH

Serial number: SPC142501CQ

MAC address information:

Operational address: f866.f214.362f

Burnt-in address: f866.f214.362f

No unicast addresses in filter

No multicast addresses in filter

Autonegotiation enabled:

Flow control restricted to: Bidirectional

Operational values:

Speed: 1Gbps

Duplex: Full Duplex

Flowcontrol: None

Loopback: None (or external)

MTU: 1526

MRU: 1526

Inter-packet gap: standard (12)

Eric, thanks Again!

xthuijs · ‎02-17-2014

Thanks for that detail Eric.

I did some further research and here is the deal:

we won't need autonego in order for flow control to work. The PAUSE frame operation is aside from the autoneg.

From the output you provided, it shows that this end wants to do bi-directional flow control and that autoneg is enabled.

Because flow control is part of the autoneg, it means taht the remote side was not willing or capable of doing flow control and therefore the operational state is set to "None".

When you disable autoneg, and our flow control operaton is bidirectional as it is right now, we will pay attention to the PAUSE frames, and we will also send them. But if the remote side is not giving us any, then there is little value to that, or if we would send one, and remote is not paying attention to it, there is also little value add there.

One other thing I wanted to mention is that PAUSE frame operation used to be useful for those NIC's that had little buffering capabilities. These days that doesnt really hold true anymore so the value of Flow Control at this level is rather limited.

Not to mention the fact that most implementations are not priority aware (extensions to the flow control standard have expanded on this however).

If you are concerned with interface loads and congestion, then probably using MQC/QOS, shaping or policing is a better option to handle congestion or using WRED to avoid congestion.

regards

xander

Eric Guo · ‎02-17-2014

Thanks very much Alex for your quick response and taking much care of this!

It is clear right now. Since we have a lots of input drop happened under the interface level and XE device side interface is showing over-subscription as well, and also there are lots of packet lost happened over this link when the traffic load is only reaching over half of this physical link limit .This is the reason, we try to enable flow-control feature between two routers.

Yes, enable QOS etc feature is best practice under this case, but I really am not sure the end user streaming expernecnie is still good or not when I input traffic shaping on upstream end.

Do you have an idea with input drop and packet lost within this link ?

thanks !

Eric

xthuijs · ‎02-17-2014

I see Eric, thanks for that rationale.

So you have the a1k as a CE and the A9k as a PE in that model? and the A1K is reporting ingress traffic drops is that correct?

Yeah, only if those input drops are caused by buffer depletion on the ingress side (doubt that) is where pause frames come in sort of handy.

Input drops can come from anywhere right, the controller, the QFP/CPP etc.

So a few things to verify are:

1) show controller gig stat on the a1k to see if there are any low level driver drops such as frame errors, crc etc.

2) check the show interface counters and see if there were ingress drops because of queue misses

CPE#show int g0/1

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 90

size shows how many there are in the queue now, drops is what we dropped because of buffer drops, flushes is what we wanted to punt to the CPU but got flushed because of SPD (selective packet discard).

3) if you also have a default queue size of 75 as above example, set it to 4096 via:

int g0/

hold-queue 4096 in and out

4) if that is all no beef, then try to verify the interface accountable QFP drops via:

show platform hardware qfp etc commands (I think is the a1k equivalent of the XR operational command of:

show controller pse qfp stat drop

5) that must result in some hint or pointer already, it may be mcast or so or some proto that can't be routed/forwarded or acl dropped maybe that can account for the input drops.

regards

xander

Eric Guo · ‎02-17-2014

Yes, you are right .

I will ask this CE end A1K device for further checking.

And in this case, my end- PE interface is also showing input drop, this number is same as Mcast number. I already ask their tech to check why we can see lots of multicast packets coming in ( we are running eBGP over this link), but they have no clue about this.

223039136 packets input, 74243589052 bytes, 23656 total input drops

23656 drops for unrecognized upper-level protocol

Received 0 broadcast packets, 23656 multicast packets

by the way, why I got below alert when I issued below command on our/PE end? is this bug of version 3.9.2

sh controllers pse qfp interface gigabitEthernet 0/2/0/23 statistics drop-summary subinterfaces location 0/2/CPU0

ERROR: sysdb_datalist FAILED. ['sysdb' detected the 'warning' condition 'A SysDB client passed a path for an item where a directory path was expected, or vice versa, or a path containing incorrectly placed '/' or reserved characters']

many thanks! Eric

xthuijs · ‎02-17-2014

hi eric,

the command show controller pse qfp is for the SIP700 in XR

the command show platform hardware qfp is for the a1k

the command show controller np counters npX loc 0/Y/cpu0 is for ethernet cards in the a9k

to find out the hardware related drops.

but that is no longer necessary, the issue is clear!

you are receiving mcast packets that we have no purpose for.

ps XR 392 is very old and not a recommended release, I would urge you to move to XR 434.

the mcast handling is much better there.

this is not a bug per-se, but you appear to be receiving mcast packets that the hardware ends up punting to the CPU for state building and if you dont have multicast routing enabled then there is no value for this.

but other things could be the reception of ospf hellos or so or any 224.x.x.x traffic that ends up getting punted.

all not a problem.

but later releases have better LPTS filters to keep the stuff away from the CPU that we have no purpose for.

regards

xander

Xander Thuijs CCIE #6775
Principal Engineer
ASR9000, CRS, NCS6000 & IOS-XR

Eric Guo · ‎02-17-2014

Hi Alex,

This is great help !

thanks a lot !

Eric

Eric Guo · ‎02-18-2014

Hi Alex,

Our customer really appreciated your help on this case. Their other ISP link has packet lost as well currently They shared some interface output to us, could you mind please take a quick check to see if everything is abnormal here ?

By the way, they are still working on identify multicast packet issue now.

Thanks very much again!

EXT-RT-18#sh controllers

TenGigabitEthernet0/0/0

0 input vlan errors

0 ingress over sub drops

TenGigabitEthernet0/1/0

0 input vlan errors

0 ingress over sub drops

GigabitEthernet0/2/0---------------------peering with our PE router

161113 input vlan errors

0 ingress over sub drops------------port-channel 100 is peering with other ISP link, packet lost happened as well.

GigabitEthernet0/2/1------------port-channel 10 is peering with other ISP link, packet lost happened as well.

0 input vlan errors

260 ingress over sub drops

GigabitEthernet0/2/2------------port-channel 10 is peering with other ISP link, packet lost happened as well.

0 input vlan errors

3220 ingress over sub drops

GigabitEthernet0/2/3------------port-channel 10 is peering with other ISP link, packet lost happened as well.

0 input vlan errors

1947 ingress over sub drops

GigabitEthernet0/2/4------------port-channel 10 is peering with other ISP link, pasket lost happened as well.

0 input vlan errors

323 ingress over sub drops

EXT-RT-18#sh controllers

TenGigabitEthernet0/0/0

0 input vlan errors

0 ingress over sub drops

TenGigabitEthernet0/1/0

0 input vlan errors

0 ingress over sub drops

GigabitEthernet0/2/0

161203 input vlan errors

0 ingress over sub drops

GigabitEthernet0/2/1

0 input vlan errors

260 ingress over sub drops

GigabitEthernet0/2/2

0 input vlan errors

3353 ingress over sub drops

GigabitEthernet0/2/3

0 input vlan errors

2150 ingress over sub drops

GigabitEthernet0/2/4

0 input vlan errors

323 ingress over sub drops

EXT-RT-18#show platform hardware qfp active statistics drop clear

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

Disabled 8921 9097269

Icmp 6 627

IpTtlExceeded 11267382 657766064

Ipv4Acl 3933578 785959704

Ipv4AclLookupMiss 3018998 558440351

Ipv4Martian 11521966 619768129

Ipv4NoAdj 3139026 370058424

Ipv4NoRoute 39646 3212618

Ipv4Null0 215993 14715497

MaxTu 7 10621

ReassTimeout 186 224056

TailDrop 898 1179567

TcpBadfrag 1 60

UidbNotCfgd 1 74

UnconfiguredIpv4Fia 751281 478895168

UnconfiguredIpv6Fia 53840 8035292

EXT-RT-18#show platform hardware qfp active statistics drop clear

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

IpTtlExceeded 91 4942

Ipv4Acl 9 606

Ipv4AclLookupMiss 8 536

Ipv4NoAdj 2 130

EXT-RT-18#show platform hardware qfp active statistics drop clear

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

IpTtlExceeded 17 1258

Ipv4Acl 4 232

Ipv4AclLookupMiss 2 120

Ipv4NoAdj 1 58

EXT-RT-18#show platform hardware qfp active statistics drop

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

IpTtlExceeded 20 1480

Ipv4Acl 18 10519

Ipv4AclLookupMiss 6 1744

Ipv4NoAdj 1 62

EXT-RT-18#show platform hardware qfp active statistics drop

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

IpTtlExceeded 32 2368

Ipv4Acl 22 12433

Ipv4AclLookupMiss 8 1876

Ipv4NoAdj 3 1222

Ipv4Null0 4 216

EXT-RT-18#show platform hardware qfp active statistics drop

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

IpTtlExceeded 75 5018

Ipv4Acl 34 13819

Ipv4AclLookupMiss 11 2062

Ipv4NoAdj 6 1444

Ipv4Null0 5 270

after PING test

EXT-RT-18#show platform hardware qfp active statistics drop

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

IpTtlExceeded 221 13133

Ipv4Acl 68 16403

Ipv4AclLookupMiss 16 3752

Ipv4NoAdj 13 2257

Ipv4Null0 7 378

EXT-RT-18#show platform hardware qfp active statistics drop

----------------------------------------------------------------

Global Drop Stats Packets Octets

---------------------------------------------------------------

IpTtlExceeded 458 27887

Ipv4Acl 114 19335

Ipv4AclLookupMiss 20 3980

Ipv4NoAdj 78 6142

Ipv4Null0 38 2052

EXT-RT-18#sh int g9 ig 0/2/0

GigabitEthernet0/2/0 is up, line protocol is up

Hardware is SPA-5X1GE-V2, address is e804.6226.7520 (bia e804.6226.7520)

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 7/255, rxload 59/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set

Keepalive not supported

Full Duplex, 1000Mbps, link type is auto, media type is LX

output flow-control is off, input flow-control is off

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:08:00, output hang never

Last clearing of "show interface" counters 3d18h

Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 233183000 bits/sec, 22438 packets/sec

30 second output rate 29110000 bits/sec, 15756 packets/sec

327256424 packets input, 256673085526 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 382812 multicast, 0 pause input

515656431 packets output, 306162835882 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

EXT-RT-18#sh int gig 0/2/0 1

GigabitEthernet0/2/1 is up, line protocol is up

Hardware is SPA-5X1GE-V2, address is e804.6226.75c9 (bia e804.6226.7521)

Description: Cogent_Link1_CCT1016381

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 10/255, rxload 127/255

Encapsulation ARPA, loopback not set

Keepalive not supported

Full Duplex, 1000Mbps, link type is auto, media type is T

output flow-control is on, input flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:14, output 00:00:10, output hang never

Last clearing of "show interface" counters 3d18h

Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

30 second input rate 501043000 bits/sec, 47246 packets/sec

30 second output rate 40979000 bits/sec, 34409 packets/sec

479375310 packets input, 562476897956 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 11741 multicast, 0 pause input

331679915 packets output, 90168694249 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Port-channel10 is up, line protocol is up

Hardware is GEChannel, address is e804.6226.75c9 (bia e804.6226.75c9)

MTU 1500 bytes, BW 4000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 12/255, rxload 135/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

ARP type: ARPA, ARP Timeout 04:00:00

No. of active members in this channel: 4

Member 0 : GigabitEthernet0/2/1 , Full-duplex, 1000Mb/s

Member 1 : GigabitEthernet0/2/2 , Full-duplex, 1000Mb/s

Member 2 : GigabitEthernet0/2/3 , Full-duplex, 1000Mb/s

Member 3 : GigabitEthernet0/2/4 , Full-duplex, 1000Mb/s

No. of PF_JUMBO supported members in this channel : 4

Last input 00:11:57, output never, output hang never

Last clearing of "show interface" counters 3d18h

Input queue: 0/1500/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/160 (size/max)

30 second input rate 2058667000 bits/sec, 194630 packets/sec

30 second output rate 177352000 bits/sec, 143475 packets/sec

2008030115 packets input, 2371096047343 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

1348271463 packets output, 359020829441 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out