flapping serial ports after IOS upgrade

[acausemaker]

I am running a 2820 router that has 2 VWIC-2MFT-T1 cards installed (64MB flash, 256MB RAM). It was running IOS ver c2800-nm-ipbase-mz.123-8.t5.bin. I upgraded it a few days ago to c2800nm-entservicesk9-mz.123-14.t2.bin. Ever since the upgrade, my ser0/0/0 (a point-to-point T1) and ser0/1/0 (frame relay) ports have been bouncing up and down. I have attached an excerpt from my syslog server. Basically, the controller goes down, which takes the serial port down, then the line protocol. If I do a show int ser br, it shows the line and proto as both "down". My ser0/0/1 line is not bouncing, but it shows a lot of CRC's/input errors/overruns.

If we plug in our old 2610 router, everything works beautifully, and I have had our ISP test their lines up to the demarc. If I revert the IOS back to the IPBASE image, everything is fine.

I even tried replacing the 2 WIC cards, and then the entire router (with an identical model & IOS version), and we're still having the same problem.

I ran HW loopback tests on the ports, and did NOT get 100% replies when I pinged each interface using the loopback plug.

The last thing I tried was setting the clock source to "line" for ser0/0/0, which didn't help - now it's set to 'internal' and so far, it hasn't crashsed yet.

I am completely baffled by this problem, is there anything else I can try to resolve this issue??

[thisisshanky]

Can you try like a 12.3.11 T (intermediate image between 12.3.8 and 12.3.14) and see if that helps!

[nchidamb]

Based on your problem description I'm suspecting that you have some clocking issue. When you had HW loopback plug did you configure 'loopback internal' under the controller where you had the HW loopback plug? When you have a normal circuit ( without loopback plug ) did you have ' network-clock-select X/Y and network-clock-participate wic '?

At the minimum you could simply have 'clock source internal' under the controller you are using. This should make the circuit / interface stable. If so, you know the problem is with your clocking config.

-Nat

[acausemaker]

I have been wondering if clocking is to blame in all this.

In my global config, I have the commands

no network-clock-participate wic 0

no network-clock-participate wic 1

I also put 'clock source internal' on the T1 0/0/0 controller (one of my troublemakers), and it went down for about 10 seconds this evening. I am still getting CRC/overruns on that interface.

If I use the network-clock-* commands, can all the interfaces have the same priority? How do I know which slot should have which priority?

thanks for the suggestions!!

[nchidamb]

In your global config you need to specify

network-clock-select 1 T1 0/0/0 <<< (priority one clock choosing from t1 0/0/0 controller)

You also need to add

network-clock-participate wic 0

network-clock-participate wic 1

The above two commands will make both wic's to sync

up with the clock that is recevied on t1 0/0/0. For

redundancy you can choose any controller as second

priority clock.

With internal clock under controller and h/w loopback plug did the interface flap stop?

-Nat

[acausemaker]

Your suggestion appears to have solved the flapping on my 0/0/0 interface, but now my 0/0/1 (the 2nd port on the same WIC card - my internet connection) is flapping up and down. My frame-relay (0/1/0) is also getting a high number of CRC/overruns, but so far has stayed up.

If I set the 0/0/1 interface to 'clock source line',will that override the network-clock-participate' command in the global config? i.e. can I force my internet connection to use the line as the clock source, since it's most likely derived from my provider anyway?

[scottmac]

There are some issues with the WICT1 V2. Depending on the rev level (Version 2 level 1.0, 1.1, or 1.2), the problem can be just the WIC (1.0), or a combination of the WIC and the IOS.

If you have the level 1.1 or 1.2, I believe it can be resolved by updating the IOS to (12.4(??)) .... The information is at my workplace (and I'm not), I'll try to find it and post it ~Monday.

The circuit is not really flapping, some element in the WICv2 "slips" and it is reported as a flap/bounce.

The symptoms exactly resemble a "clock slip" where there is a sync mismatch and the circuit periodically drops until sync is re-established.

The folks at work have seen this problem on 1750s and (with a slight variance in symptoms) the X800 series ISR routers.

FWIW

Scott

[acausemaker]

I changed the clock source to internal for my 2 interfaces that are bouncing, and we are still having the same problem.

I have tried enabling the network-clock-participate wic 0 & 1, and setting them to synch with my t1 0/0/0 controller. I am still getting CRCs/overruns.

During the HW loopback tests, I did set the clock source to 'internal', and some of the pings still failed (mainly the 0xFFFF data type).

The next thing I am going to try is a new IOS image, that's about the only thing left to try.

[scottmac]

Here is an excerpt from an email from one of my co-workers regarding this problem:

(quote)

If you are having a problem with a circuit bouncing and the circuit tests clean or the Telco tells you that the CPE end of the circuit is losing sync, check the show service-module command on the Cisco router for path and line code violations. If you are having path and line code violations then you will need to check the WIC module with the show diag command.

If the WIC-module is WIC-1DSU-T1-V2 and the hardware revision is:

1.0 - RMA the WIC card and upgrade the IOS version to a version greater than 12.3(8)T7

1.1 - Upgrade the IOS version to a version greater than 12.3(8)T7

1.2 - Upgrade the IOS version to a version greater than 12.3(8)T7

Version 12.4(1a) is preferred if there are no ATM interfaces and the Flash and RAM will support it.

The hardware revision 1.0 of the version 2 WIC is known to have a problem but some versions of IOS also cause the CPE to lose sync and cause the circuit to appear to bounce.

The bug ID for the IOS is CSCsa55375, it specifies only the 2800 series routers but I have confirmed that it affects the 3745 and the 1760 routers also

(end quote)

Hope this helps

Scott

[acausemaker]

Scott,

thank you for the info. I am going to try upgrading to the 12.4(1a) tonight and cross my fingers. That is not the part number on my WIC card, but my HW revision is 1.0, and at this point I will try about anything to get these lines stable.

thank you again.

[clsalaza]

Hi,

I've been reading your post and coincide with the others about the clocking.

Some information about the disscution:

1) vwic-2mft is a double T1/E1 card with just one PLL on card. That means that the card by itself is able to attach just the one clock eventhough it has capacity for 2 T1s. The fact that the card has just one PLL is due to the space limitation. To overcome that problem AIM module and other solutions can be use but if a NM is used the the limitations of the NM come in to play too. Having just one PLL does not mean that the card is not able to handle 2 T1s, but just that can handle one clock by the 2T1s.

2) Every IOS handle differently the subroutines for the master clock election.

now that we are done my observs.

With the old IOS if you do a "show controller t1" independently that you selected a clock source for the channel groups who appears as master clock and what's the master clock the other controllers are pointing to? with the new IOS do s show controller t1 again and compare de output of the clock lines against the old one do they coincide?? chances are that with the old IOS the vwic was selecting a different port to be the master than the one is being selected now. You can play around with the clock primary command and adjust it, but still have some limitations, in your case it should work since it was working previously (unless your telco had change anything in their cloking network since then).

If you want to can collect the files of the show controllers t1 so we can take a look.. cheers