Realibility drop and input error increase at Multilink Interface

[muhammad shahrir osman]

Hi All Cisco and Network Expert,

Need some advise on the current issue that I'm facing right now.

The problem is that my customer observes input error,frame error and realibilty drop at only Multilink Interface. No error detected at the serial interface.

Current network setup is the Service Provider is using Juniper router thru E1 then using NTU connected to my customer router (Multilink with 2 serial interface - 2x2Mbps). Service provider have done intrusive test on the physical part and no error detected.

I've tried google the problem and found out that few people facing the same problem but did not find any solution to this. In Cisco support, only https://supportforums.cisco.com/message/827055?tstart=8   have provided some clue to this problem but unfortunately I do not have previllage to access http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsj26883 to further understand the problem.

Below is some of the testing result that I've done with customer.

1)  I configured the Multilink interface with only 1 serial interface. For testing, I pumped high traffic and only Multilink interface having error and realibility drop. No issue with serial interface.

Router#sh int mu1   

Multilink1 is up, line protocol is up

  Hardware is multilink group interface

  Description: *** xxxxxxxx***

  Internet address is 58.x.x.162/30

  MTU 1500 bytes, BW 4096 Kbit/sec, DLY 20000 usec,

     reliability 254/255, txload 121/255, rxload 17/255

  Encapsulation PPP, LCP Open, multilink Open

  Open: IPCP, loopback not set

  Keepalive set (10 sec)

  DTR is pulsed for 2 seconds on reset

  Last input 00:00:00, output never, output hang never

  Last clearing of "show interface" counters 00:13:09

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 472/600 (size/max)

  5 minute input rate 282000 bits/sec, 53 packets/sec

  5 minute output rate 1949000 bits/sec, 196 packets/sec

     47597 packets input, 30642805 bytes, 0 no buffer

     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

     975 input errors, 0 CRC, 975 frame, 0 overrun, 0 ignored, 0 abort

     154570 packets output, 192428867 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

     0 carrier transitions

2) Same test result if I bundled the 2 serial interfaces with Multilink, only the Multilink having issue and both serials not having problem.

Router#sh int mu1

Multilink1 is up, line protocol is up

  Hardware is multilink group interface

  Description: *** xxxxxxxx***

  Internet address is 58.x.x.162/30

  MTU 1500 bytes, BW 4096 Kbit/sec, DLY 20000 usec,

     reliability 253/255, txload 5/255, rxload 5/255

  Encapsulation PPP, LCP Open, multilink Open

  REQsent: CDPCP

  Open: IPCP, loopback not set

  Keepalive set (10 sec)

  DTR is pulsed for 2 seconds on reset

  Last input 00:00:00, output never, output hang never

  Last clearing of "show interface" counters 12:26:39

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 89000 bits/sec, 24 packets/sec

  5 minute output rate 92000 bits/sec, 23 packets/sec

     215677 packets input, 156087189 bytes, 0 no buffer

     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

     1472 input errors, 0 CRC, 1439 frame, 0 overrun, 33 ignored, 0 abort

     213659 packets output, 159653273 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

     0 carrier transitions

Any idea what has caused this problem? I've check the configuration and I don't think there is any problem with it.

Configuration snippet:

!

interface Multilink1

description *** xxxxxxxx***

bandwidth 4096

ip address 58.x.x.162 255.255.255.252

no ip redirects

no ip proxy-arp

ip nat outside

ip virtual-reassembly

ppp multilink

ppp multilink links minimum 1

ppp multilink load-threshold 1 outbound

ppp multilink group 1

no cdp enable

hold-queue 600 out

!

interface Serial0/0/0

bandwidth 2048

no ip address

encapsulation ppp

ppp multilink

ppp multilink group 1

no fair-queue

invert txclock

!

interface Serial0/0/1

bandwidth 2048

no ip address

ip nat outside

ip virtual-reassembly

encapsulation ppp

ppp multilink

ppp multilink group 1

no fair-queue

invert txclock

!

Thanks in advance for any advise and help. Appreciate it! Thank you

Regards,

Arel

[John Blakley · Accepted solution]

Can you post "sho int s0/0/0" and "sh controllers"?

[muhammad shahrir osman]

Hi Blakley,

Thanks for the feedback.Sorry I've mistakenly click "the correct answer" button....

Here is the output that requested.Please also see attacment. Thank you!

.

sh int Serial0/0/0

Serial0/0/0 is up, line protocol is up

  Hardware is GT96K Serial

  MTU 1500 bytes, BW 2048 Kbit/sec, DLY 20000 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation PPP, LCP Open, multilink Open

  Link is a member of Multilink bundle Multilink1, loopback not set

  Keepalive set (10 sec)

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters 5d06h

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 1 packets/sec

  5 minute output rate 0 bits/sec, 1 packets/sec

     118903 packets input, 18839359 bytes, 0 no buffer

     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

     122857 packets output, 15234942 bytes, 0 underruns

     0 output errors, 0 collisions, 42 interface resets

     2638 unknown protocol drops

     0 output buffer failures, 0 output buffers swapped out

     0 carrier transitions

     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up

[John Blakley]

I don't see anything wrong with them. What version IOS are you running?

[muhammad shahrir osman]

Hi,

I post again here at my thread. Thank you

#sh version

Cisco IOS Software, 1841 Software (C1841-IPBASE-M), Version 15.0(1)M4, RELEASE SOFTWARE (fc1)

[John Blakley]

Okay...let's work off of this one

The bug report only shows 12.x versions and doesn't touch on the 15.x version. You may want to get a TAC case created if possible. I pasted the report for you to look at in here:

sh int/ on MLP Bundle interface shows 100% input errors

Symptom:  on a multilink bundle interface, the output from "show interface" will show an input error count which includes essentially 100% of the traffic being received.  Other drop counters, usually the "bad frame" counter, are also incremented to match.  For example:# show interface Multilink1 Multilink1 is up, line protocol is up Hardware is multilink group interface Internet address is 10.20.0.1/30 MTU 1500 bytes, BW 3072 Kbit, DLY 100000 usec, reliability 181/255, txload 1/255, rxload 1/255 Encapsulation PPP, LCP Open, multilink Open ... 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 6080 packets input, 1027997 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 6077 input errors, 0 CRC, 6077 frame, 0 overrun, 0 ignored, 0 abort ^^^^                      ^^^^Impact:  Very Low.  This is an accounting error; the counters are being accidentally incremented each time Multilink begins to receive a new datagram.  This does not affect actual traffic flow or operation of the bundle, it is only the statistics that are in error.Workaround: none.

[muhammad shahrir osman]

I think i'll advise customer to open TAC case to further investigate.

Erm...but from the report it said the impact is very low and no workaround? Do you think so?

At the first place my customer is complaining about the throughput issue. The bandwidth did not meet as expected ~4Mbps (in my customer case). After several testing, then I detected this Multilink error. So I think this could be the issue.

Thanks a lot for helping. Thank you.

[John Blakley]

If they have the means to open a TAC case, it wouldn't hurt to find out if there is a known issue with 15.x. They can add it to the list. The bug states that it's not performance impacting. If you're having speed issues and can't hit your CIR, I'd contact the SP and have them look at their ML interface on their end to see if they have something misconfigured.

One question though...why do you have a bandwidth setting of 2048 on a serial link? Is this a T1? It's also recommended not to put the bandwidth statement on the multilink. The reason for this is so the ML interface can adjust should one of the links go down. If you hard code the bandwidth statement, your bandwidth wouldn't change from the 4Mb that you have even though half of your link went down. This obviously would cause issues with calculations in routing protocols and QoS.

[muhammad shahrir osman]

It is an E1. I've checked with SP and they said no issue with configuration at their end as all other customer that using Multilink is using the same and standard configuration. For bandwidth configuration, I think at my customer router has been configured since the first installation as per advise from SP.

Thanks for your advise Sir.

[saurab mattoo]

Was this issue resolved. I am having same issue right now. No errors on 4 T1 serial interfaces but multilink is showing loads of errors... Had you tried replacing the cables on serial interfaces?