ATM problem with various models

ronald.tuns · ‎07-11-2011

Hi everyone,

I'm having a problem with various Cisco ADSL modems and one specific ISP. The ISP is changing their ATM network to Ethernet and ever since the Cisco routers regularly lose their connection. Only resetting the modem brings the line back. According to the ISP this is a known problem with Cisco modems and they have sent an example config to configure the ATM.

We have implemented this config, but the problem is still there. Since the problem does not occur with modems from other brands (Alcatel and even Linksys!) the ISP lays the responsibility with us, being the supplier of the modem.

According to the ISP oam PVC managed causes the problem, only this is already disabled in our configs.

I have enabled ATM debugging on one router (887 ISR) and this error occurs repeatedly:

cisco ATM_UCC_GetChannelHandler_PQ2CE: wrong ChannelCode

Any idea what this could mean ?

Thanks for any help !

Regards,

Ronald Tuns

Frederic Vanderbecq · ‎07-14-2011

Hello Ronald,

looking to the error, it could be linked to the following bug:

CSCtl72890 duplicate of CSCtq05004 Dialer loses IP Address sporadically

CSCtq05004 is currently not fixed. Though, with the description above, it is not enough to say for sure this is the issue you are facing.

As this may be complex to resolve via the forum, it might be best to raise a TAC case for this issue.

Things that could be interesting to log at the time of the problem is:

- debug atm error

- debug atm event

((the above debugs could be enabled prior to the problem appearance and their logs checked at the time of the probmem))

- show atm pvc / a few times in a row

- show dsl int atm ... or show controller dsl ...

At the moment, what is the configuration used on the CPE ? Maybe the configuration could shed additional light to this matter.

paolo bevilacqua · ‎07-14-2011

You need to update ADSL firmware. That unfortunately requires a support contract for download from cisco.com.

ronald.tuns · ‎07-18-2011

Hi everyone,

Thanks for your replies.

Most models have the latest IOS version, which should include the lastest ADSL firmware ?

Furthermore, I have found that if I downgrade a 877 ISR to IOS version 12.3-8.YI2 the connection is stable. Suffice to say I want to use at least 12.4 so I am affraid that this is going to be a trial-and-error to find the right IOS version ?

I haven't tried a different IOS on The 800 series ISRs yet.

I am considering opening a TAC case, since it is definitely IOS related.

Regards,

Ronald

paolo bevilacqua · ‎07-18-2011

Latest IOS what ?

You need to check "show dls interface".

If the ADSL FW is behaving with you particular line and DSLAM, then you have a stable connection is stable. Otherwise, you do not.

That is mostly indipendent from IOS.

inaxeon123 · ‎08-01-2011

If you're having the same problem as me, which it appears it's possible that you may be, then you want to steer clear of the "latest" IOS.

I've found this on the 15.0.x branch. Specifically, the issue arrives in early 2011 with 15.0(1)M5. According to my testing, 15.0(1)M4 is the last good release, which was built November 2010.

I guess it's possible that this years' 12.4 releases have had this code merged into them too.

Although our circumstances are all quite different, to me this all points back to one thing: Broken ATM code. But surely it's not possible that Cisco would produce fundamentally broken releases for 6 months?

Here's a potentialy related issue from another user, with some commentary on my scenario:

https://supportforums.cisco.com/thread/2092730

ronald.tuns · ‎08-02-2011

Hi Matthew,

Thanks for your input! I am baffled too.

This problem is haunting me on more and more connections, all which worked fine until the ISP adjusted their ATM signal because of their migration.

Try to explain to your customers that a 30 dollar low-end router delivers a perfect stable line and the router they bought at 10 times that price can't do that...

I tried various 15.x releases all with the same error and the only 12.4.x that is stable, stalls the download to all but a stop.

That is why I have opened a TAC to see what comes up, so far it's a trial-and-error process though.

I'll keep you posted on my findings...

paolo bevilacqua · ‎08-02-2011

If you want help from this forum, post config, and "show dsl interface".

inaxeon123 · ‎08-02-2011

I'm not entirely sure if me posting 1000 lines of config to this thread is going to add an awful lot of value to this discussion at present. But OK, here's some of it:

Router#show version

Cisco IOS Software, C870 Software (C870-ADVIPSERVICESK9-M), Version 15.0(1)M4, RELEASE SOFTWARE (fc1)

...

Router#show dsl interface atm0

ATM0

Alcatel 20190 chipset information

ATU-R (DS) ATU-C (US)

Modem Status: Showtime (DMTDSL_SHOWTIME)

DSL Mode: ITU G.992.5 (ADSL2+) Annex M

ITU STD NUM: 0x03 0x2

Chip Vendor ID: 'STMI' 'BDCM'

Chip Vendor Specific: 0x0000 0x918F

Chip Vendor Country: 0x0F 0xB5

Modem Vendor ID: 'CSCO' ' '

Modem Vendor Specific: 0x0000 0x0000

Modem Vendor Country: 0xB5 0x00

Serial Number Near: FCZ1426C2HW

Serial Number Far:

Modem VerChip ID: C196 (3) capability-enabled

DFE BOM: DFE3.0 Annex M (3)

Capacity Used: 97% 99%

Noise Margin: 5.5 dB 6.0 dB

Output Power: 19.0 dBm 12.5 dBm

Attenuation: 41.0 dB 23.0 dB

FEC ES Errors: 87 0

ES Errors: 10 10

SES Errors: 8 0

LOSES Errors: 8 0

UES Errors: 0 289

Defect Status: None None

Last Fail Code: None

Watchdog Counter: 0x1E

Watchdog Resets: 0

Selftest Result: 0x00

Subfunction: 0x00

Interrupts: 4152 (0 spurious)

PHY Access Err: 0

Activations: 1

LED Status: ON

LED On Time: 100

LED Off Time: 100

Init FW: init_AMR-4.0.017_no_bist.bin

Operation FW: AMR-4.0.017.bin

FW Source: external

FW Version: 4.0.17

DS Channel1 DS Channel0 US Channel1 US Channel0

Speed (kbps): 0 10847 0 2048

Cells: 0 29305350 0 553592904

Reed-Solomon EC: 0 12049 0 0

CRC Errors: 0 445 0 13

Header Errors: 0 434 0 77

Total BER: 0E-0 6703E-10

Leakage Average BER: 0E-0 3957E-9

Interleave Delay: 0 24 0 58

ATU-R (DS) ATU-C (US)

Bitswap: enabled enabled

Bitswap success: 0 0

Bitswap failure: 0 0

LOM Monitoring : Disabled

DMT Bits Per Bin

000: 0 0 0 0 0 0 0 7 9 A B B C C D C

010: D D D D D D D D D D D D D D D D

020: C B A A A A A 9 9 9 9 9 9 9 9 8

030: 8 8 8 8 8 7 7 7 0 0 0 0 A B C C

040: C C C D D D D D D D D C D D D D

050: D C C C D C D C C C C C C C 2 C

060: C C C C C C C C C C C C C C D C

070: C C C C C C C B B B B B B B B C

080: B 7 8 B B B B B B B B B B B B B

090: B B B B B B B B B B B A B B B B

0A0: A A A B B A B 6 A A A A A A A A

0B0: A A A A A A A A A A A A A A A B

0C0: A A 9 A A A A A 9 9 A A A A A A

0D0: A 9 8 5 9 9 A A A A A 9 9 9 9 9

0E0: 8 4 5 8 9 9 9 9 9 9 9 9 9 9 9 9

0F0: 2 8 9 9 7 9 9 9 9 9 8 8 7 6 6 9

100: 8 8 9 9 9 9 9 8 9 8 9 8 9 9 8 9

110: 8 8 8 9 8 8 8 8 8 7 5 8 8 8 8 8

120: 8 7 8 8 8 8 7 8 8 8 8 8 8 8 7 7

130: 7 7 7 6 7 7 7 7 7 7 7 7 7 7 6 0

140: 5 6 6 6 6 5 5 5 5 5 0 5 5 5 0 5

150: 5 3 0 4 0 5 5 5 5 5 5 5 5 5 5 5

160: 4 4 5 4 4 0 4 0 4 0 4 4 4 4 4 4

170: 3 0 0 3 3 3 3 3 2 3 2 2 3 2 2 2

180: 2 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2

190: 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

1A0: 2 2 2 2 0 2 0 1 1 1 1 1 1 1 1 1

1B0: 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0

1C0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1D0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1E0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1F0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

DSL: Training log buffer capability is not enabled

And some relative snips of config:

!

interface ATM0

no ip address

no atm ilmi-keepalive

!

interface ATM0.1 point-to-point

atm route-bridged ip

bridge-group 3

pvc 0/101

oam-pvc manage

encapsulation aal5snap

!

interface BVI3

ip address 78.105.x.2 255.255.240.0 secondary

ip address 78.105.x.1 255.255.240.0

ip nat outside

ip virtual-reassembly

!

interface Vlan3

no ip address

bridge-group 3

!

None of the rest of my config has anything to do with ATM/DSL or anything else to do with the external interface.

I'm not too concerned about this issue at present. The main issue I guess was that I'm considering upgrading to a 1921 at some point, which is going to require me running newer versions of IOS, which had better darn well work!

ronald.tuns · ‎08-03-2011

I'll post the ATM errors I get:

ATM_UCC_GetChannelHandler_PQ2CE: wrong ChannelCode

003006: *Jul 28 07:48:05.951 PCTime: ATM0.1(O):

VCD:0x1 VPI:0x2 VCI:0x20 DM:0x100, MUXETYPE:0x0009 Length:0x12

inaxeon123 · ‎08-03-2011

Yep. I (more or less) got that message at the exact point the connection died.

Mine was (something like):

ATM_FCC_GetChannelHandler_PQ2CE: wrong ChannelCode

Yours:

ATM_UCC_GetChannelHandler_PQ2CE: wrong ChannelCode

OK so it's slightly different to yours, but I don't think this matters because the CPU in the 877 is MPC8272, which has no UCCs, so instead the DSL chips are connected to an older style FCC, so I'd fully expect this small difference in output

Given that we're not running the same code at this lower layer, this to me, points an even fatter finger at the higher level ATM code.

paolo bevilacqua · ‎08-03-2011

Matthew Millman wrote:

interface ATM0

no ip address

no atm ilmi-keepalive

!

interface ATM0.1 point-to-point

atm route-bridged ip

bridge-group 3

pvc 0/101

oam-pvc manage

encapsulation aal5snap

!

interface BVI3

ip address 78.105.x.2 255.255.240.0 secondary

ip address 78.105.x.1 255.255.240.0

ip nat outside

ip virtual-reassembly

!

interface Vlan3

no ip address

bridge-group 3

You have a mixed-up incorrect configuration. With "atm route-bridged ip" you can have IP address and NAT statement directly under atm subinterface, does not need a BVI interface and does not need IRB statements. If doing so, remember toalso change nat statements accordingly.

Otherwise, "atm route-bridged ip" is not needed. That may be the case where you have hosts with a public address on VLAN3.

Note, I don't claim these changed will fix your issue, but if you have to go to the TAC with a bug report, you must go with a clean configuration.

inaxeon123 · ‎08-03-2011

VLAN3 acts as a poor-mans DMZ. It has a number of hosts with public IP addresses.

I've removed the "atm route-bridged ip" statement, which was a hangover from and older setup.

I'm pretty skeptical, but not too arrogant to give it another go. Next time I can afford some downtime, I'll boot it up with 15.1 again with that statement removed, see what happens.

inaxeon123 · ‎08-07-2011

OK-- Tried 15.1(4)M1 and 15.0(1)M5 without "atm route-bridged ip" The VC lasted all of about 15 seconds. So that wasn't the problem.

I can now say that an easy way to see if the release you're running has this problem is within the "show dsl interface atm0" command.

Releases which output this:

Serial Number Near: FCZ1426C2H 877-M-K9 15.1

Are bust.

Releases which output this:

Serial Number Near: FCZ1426C2HW

Are OK.

ronald.tuns · ‎08-08-2011

I have tried various 12.x and 15.x IOS on both 870 and 880 models and that doesn't improve at all.

Like I said, I have opened a TAC and that's not going all too well. Tried different IOS and DSL firmwares, all with the same result: BadChannel errors and a drop of the line.

I'm wondering where this is going...'cause it looks like a needle in a haystack.

Small update:

The ISP has lowered the speed of the connection from 8 to 6 Mb. Also I have uploaded c880data-universalk9-mz.150-1.M7.bin as advised by the Cisco tech working on this case.

Unfortunately it's not possible to pinpoint which change made the difference, but the line is in sync for 24 hours now, the first time since the problem arose.

Crossing my fingers...