Traffic flapping in regular intervals

[tharakasj]

We are telecom provider. Edge router is Cisco 7604 with SIP 200 cards

Core Router is cisco 12406. We have a MPLS network. router were connected using

STM1 link. Link is terminated by POS interface of those routers. the issue is that every day traffic flapp in regular intervals (15minuts) from 8.35am to 5pm.

please advice is that virus issue or link issue or router issue.

Anyway no errors in links and router interfaces.

In every flapping time, few packets drops and delays go over 1000ms. but normal delay is 2ms when pingging between routers.

Images is attached .

Please help us to resolve this

[Giuseppe Larosa]

Hello Tharaka,

first of all you have to find out if what you see in SNMP derived graphs is real or not.

if routing protocol are stable ( show ip ospf neig det or show isis neigh detail provide the uptime) the problem may not be real.

What kind of counters are used to build the graph?

the standard 32 bit counters ifincoctets ifoutoctets are not able to deal with speeds like STM-1 or greater.

At full speed an STM-1 causes wrapping of the standard byte counters in less then 4 minutes. (229 seconds)

(2^32 is roughly 4 billions = 4294967296    150 Mbps is 18750000 Bps byte per second  divide the first by the second and you get 229 seconds.)

The problem is called wrapping of counters.every N minutes the counter goes in overflow and the calculation of speed rate is made by making the difference between last two reading of octets variable. So every N minutes depending on average bit rate the counter make an hop that can appear in the graph as a minimum.

A solution to this problem is the use of 64 bit counters that contain HC in the prefix they are called

ifHCinoctets and ifHCoutoctets these are able to represent 2^64 range of values.

You should verify if the C7600 supports the HC version of the counters, if so the solution is to poll these other MIB variables.

C12000 should support the HC counters it was supported on first version of C12000 in 1999.

Hope to help

Giuseppe

[tharakasj]

Thanks for your comments.

But we check SNMP using MRTG and PRTG.further we notice that every flap, ping delays go beyond 1000ms with few packet drops. otherwice it is 2ms. IS-IS protocol logs , hols time expired and new ajacency created/!!!!.

Our customers also expericancing this traffic pattern and actucally they are suffering. So traffic delay is expericncing by us.

please comment.

[Giuseppe Larosa]

Hello Tharaka,

the problem is real if ISIS flaps.

I agree I had missed you had mentioned the increased delay

I would say that the problem may be  probably in the SDH world and it is probably related to clocking or the way the VC are carried in the SDH hierarchy.

Are you using default timers for ISIS including for max-age?

Hope to help

Giuseppe

[tharakasj]

Hello, Thanks for support.

We increased halding time upto 120s. Now IS IS log is not over there. But still issue is exist.

How change cloking in SDH. is it in router or have to change those with STM receiver and all.

Please respond.

Thanks

[Giuseppe Larosa]

Hello Tharaka,

may you post ( if supported)  (where posx/y/z is your pos interface)

a

show controller pos x/y/z

+

show interface pos  x/y/z

to see if there are errors at SDH level like severly errored seconds (SES) and so on

Hope to help

Giuseppe

[tharakasj]

Hello,

Please refer to attachment.both sh cont is attached.

Thanks you very much

Cisco 12406 core Router

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

POS2/2/2

SECTION

  LOF = 0          LOS    = 0                            BIP(B1) = 1096

LINE

  AIS = 2425       RDI    = 57         FEBE = 4296782    BIP(B2) = 37334688

PATH

  AIS = 2376       RDI    = 2323       FEBE = 1814026    BIP(B3) = 11892092

  LOP = 2530       NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None

Active Alarms:  None

Alarm reporting enabled for: SF SLOS SLOF B1-TCA B2-TCA PLOP B3-TCA

Framing: SDH

APS

  COAPS = 6677       PSBF = 3554     

  State: PSBF_state = False

  ais_shut = TRUE

  Rx(K1/K2): 00/00  S1S0 = 02, C2 = 16

  Remote aps status (none); Reflected local aps status (none)

CLOCK RECOVERY

  RDOOL = 0        

  State: RDOOL_state = False

PATH TRACE BUFFER : STABLE

  Remote hostname : GPAS-MEDG-004          

  Remote interface: POS3/2/0     

  Remote IP addr  : 10.13.1.2      

  Remote Rx(K1/K2): 00/00  Tx(K1/K2): 00/00

BER thresholds:  SF = 10e-3  SD = 10e-6

TCA thresholds:  B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

Optical Power Monitoring

Module temperature    =  26.968 C

Transmit Bias Current =  3548 uAmp

Tx Output Power       =  -10 dBm  (750 in units of 0.1 uW)

Rx Output Power       =  -11 dBm (711 in units of 0.1 uW)

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Edge router -cisco 7604

[Giuseppe Larosa]

Hello Tharaka,

there are errors on the C12000 side.

for BIP errors explanation have a look at

http://www.cisco.com/en/US/tech/tk482/tk607/technologies_tech_note09186a0080094a79.shtml

about clocking see

http://www.cisco.com/en/US/tech/tk482/tk607/technologies_tech_note09186a0080094bb9.shtml

if the two routers are connected back to back one of the two devices has to act as DCE providing clock

one side needs the command

clock source internal

the other side should be in clock source line

Hope to help

Giuseppe

[tharakasj]

Hello,

Can we change one side in to INTERNAL when the router is sending traffic and in productions state.

Are there any risk my friend.

Prageeth

[Giuseppe Larosa]

Hello Tharaka,

can you confirm that the two routers are connected in back to back without any SDH device in the middle?

if so one device needs to be configured with clock internal. The impact should be limited you can ask for a maintenance time window for safety but it should be less of what happens every 15 minutes during daytime.

The interface may flap up/down this should be the maximum impact but it shouldn't.

Hope to help

Giuseppe

[tharakasj]

Dear Giuseppe,

Thanks for your help.

No SDH device in middle.

in this case do we need to setup Interal - Line mode.

Thanks

Prageeth

[Giuseppe Larosa]

Hello Prageeth,

one side must be with clock internal, according to documentation both is allowed.

You need to avoid to have both devices configured with clock source line because it causes clocking problems in back to back scenario.

The side with clock internal will become the clock reference for the link this is the result.

Change one side to internal in a maintenance window and wait some time to see if your troubles are fixed.

Hope to help

Giuseppe

[tharakasj]

Hello Giuseppe,

We have done the changes as you describe. But still flapping is there. Does it need shutdown and no shutdown to apply the clock setting.

Waiting your feedback.

Thanks

Prageeth

[Giuseppe Larosa]

Hello Prageeth,

please post the configuration of the POS interfaces both sides and

show interface pos

show controller pos

There may be something else to tune like payload scrambling

Hope to help

Giuseppe

[tharakasj]

Hello Giuseppe,

hearwith attached sh int pos .. please see it!!

My friend, 1st i change clock to Internal at Cisco 7604 side. issue is exist. but no any alarms generated.

then 2nd  i set to original and change clock to Internal on Cisco 12406 side. then issue is exist and many alarms were generated (transmmission alarms). and traffic goes down for a 5 minute time.So we set all to original state.

But 1st case we didnt reset port (shutdown and up).

Thanks for your help

Prageeth