Cisco 888 modems in back to back configuration - CPE dropping packets

[ChuckHaynes]

Greetings,

We have 4 sets of 888 DSL modems running in back to back configurations. Recently, the CPE side on three sets started dropping packets and we can't figure out why. One set of modems is still running fine (which makes no sense as they all had the same config and the working set is going into the same switch as one of the non-working sets). The CO sides are fine, but the CPE sides are dropping about 30%. I (nor any of my fellow co-workers) are aware of any network changes, as we would be the ones to do it. When I login to a CPE and do a 'sh int', I can see that ATM0 is dropping packets from an unknown protocol. I'm not sure if that is part of the problem or not. I have pasted the config below. Any ideas and/or suggestions would be greatly appreciated. BTW, the standard subnet/vlan here is .47 and the secondary is .43. It appears the problem doesn't exist if I change the IP addresses of the modems to .47 and swap the switch port to vlan 47, but we really need them on 43. The odd thing, it appears that once you frist config a new set of these, they work fine for a little while (several minutes) and then the CPE side starts dropping. Not only does it drop packets, but the response time goes from 7-8ms to 95ms. I'm not sure if there is a routing issue or what.

version 15.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname SMK_UG_DSL6_BrCPE
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
memory-size iomem 10
!
!
!
!
!         
!         
!         
!         
!         
!         
ip cef    
no ipv6 cef
!         
!         
multilink bundle-name authenticated
license udi pid CISCO888-K9 sn FTXxxxxxxxx
!         
!         
username xxxxxxx privilege 15 password 7 xxxxxxxxxxxxxxxxxx
!         
!         
!         
!         
!         
controller DSL 0
 mode atm
 ignore-error-duration 30
 line-rate 1024
!         
!         
!         
!         
!         
!         
bridge irb
!         
!         
!         
!         
interface BRI0
 no ip address
 encapsulation hdlc
 shutdown
 isdn termination multidrop
!         
interface ATM0
 no ip address
 no ip route-cache
 no atm ilmi-keepalive
!         
interface ATM0.1 point-to-point
 no ip route-cache
 bridge-group 1
 pvc 0/35
!        
!         
interface FastEthernet0
 no ip address
!         
interface FastEthernet1
 no ip address
!         
interface FastEthernet2
 no ip address
!         
interface FastEthernet3
 no ip address
!         
interface Vlan1
 no ip address
 bridge-group 1
!         
interface BVI1
 ip address 10.110.43.193 255.255.255.0
!         
ip forward-protocol nd
no ip http server
no ip http secure-server
!         
!         
ip route 10.0.0.0 255.0.0.0 10.110.43.254
ip route 192.168.0.0 255.255.0.0 10.110.43.254
!         
!         
!         
control-plane
!         
bridge 1 protocol ieee
bridge 1 route ip
!         
!         
line con 0
 no modem enable
line aux 0
line vty 0 4
 login    
 transport input all
!         
scheduler max-task-time 5000
!         
end

[Leo Laohoo]

Kindly post the output to the command "sh interface atm <BLAH>".

Are the dropouts happening all the time or only during certain hours of the day?  If it is certain hours of the day, around what time (local)?

[ChuckHaynes]

The dropouts happen constantly (all day long). Here is the output from 'sh int atm0'

ATM0.1 is up, line protocol is up
  Hardware is MPC ATMSAR, address is c067.afc9.8378 (bia c067.afc9.8378)
  MTU 4470 bytes, BW 1024 Kbit/sec, DLY 360 usec,
     reliability 255/255, txload 1/255, rxload 255/255
  Encapsulation ATM
  Keepalive not supported
     49493419 packets input, 252784652 bytes
     70001 packets output, 4211439 bytes
     0 OAM cells input, 0 OAM cells output
  AAL5 CRC errors : 0
  AAL5 SAR Timeouts : 0
  AAL5 Oversized SDUs : 0
  Last clearing of "show interface" counters never
SMK_UG_DSL5_BrCPE#sh int atm0  
ATM0 is up, line protocol is up
  Hardware is MPC ATMSAR, address is c067.afc9.8378 (bia c067.afc9.8378)
  MTU 4470 bytes, sub MTU 4470, BW 1024 Kbit/sec, DLY 360 usec,
     reliability 255/255, txload 1/255, rxload 255/255
  Encapsulation ATM, loopback not set
  Keepalive not supported
  Encapsulation(s): AAL5
  20 maximum active VCs, 1024 VCs per VP, 1 current VCCs
  VC Auto Creation Disabled.
  VC idle disconnect time: 300 seconds
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Per VC Queueing
  5 minute input rate 1580000 bits/sec, 1085 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     98322532 packets input, 829045377 bytes, 0 no buffer
     Received 48922030 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     70018 packets output, 4213702 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     8165 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

reliability 255/255, txload 1/255, rxload 255/255

Hmmmm ... This means that the ATM0 interface is downloading something a full bandwidth.  255/255 means 100%.    Look at the "packets input" vs "packets output".  There's a significant ratio of data going from the ISP to your router.  
 

I'd run a netflow report so you'll be able to determine what clients are actually pulling data from the WAN.  

[ChuckHaynes]

Let me shed a little light on the network topolgy. The data circuits are provided by Frontier and the ISP is AT&T. Here is a diagram of the layout.

Router-->Main Switch-->DSLco------------DSLcpe

There is nothing connected to the DSLcpe side (we were trying the process of elimination). The cable between the co and cpe is like 1-foot right now. Of course the network (in it's entirety) is was more complex than the diagram above, but we moved the modems directly to the main switch without anything behind them and the problem still exists. Note - If I wasn't clear, these modems are just used as basic Ethernet extenders to provide a network connection to an area that doesn't have fiber.

[Leo Laohoo]

Just check your interface.  You shouldn't be getting your download pipe getting 100% all the time.  Check the interface status regularly.  If the result is 255/255 in either "Tx" or "Rx' then something is choking your bandwidth and this is indicative of your packet loss.  

If this is the case, then the first step could be to enable netflow so you'll be able to determine who your "top talkers" are and what their destination address(es) is/are.  

Another harsh method is to disconnect the LAN cable one by one until the ping times drop.

[ChuckHaynes]

I finally figured out the problem. It was the "line-rate" command that caused the issue. Previously, I have set it at 1024 when we were dealing with some dirty lines/noisy conditions and all was good. I have no idea what changed, but now leaving it on 1024 will allow the problem to occur. If you set it as line-rate auto, the problem gets worse. The packet loss still occurs, but the ping times jump to around 179ms. Now here's the weird part... If you remove the line-rate command from the config and let it auto detect, it will slect the rate of 2304 (which is the highest rate for a 2-wire connection). Strange that 2304 works fine but 1024 (which is slower, so you think it would be more stable), doesn't work. What's even stranger is the default should be line-rate auto, but that makes it even worse. You have to actually remove the command totally for it to work correctly? Now as to why the problem just started out of the blue... We had one set that never developed the problem (it's running IOS 15.0). We had two sets that did develop the problem (they are running IOS 15.2). Given this information, I thought that maybe this "line-rate" issue was a bug in 15.2. The other weird part is that we had a third set that developed this issue as well. It is running 15.0 (because I checked this early on to potentially elimate an IOS issue). However, after finally figuring out that the line-rate issue was the problem, I went back to check the version on the last set. The CO was sure enough 15.0 as I thought, but the CPE is deep underground (disconnected now). I will have to wait until a later date to see which version it is. It's possible that since the devices were mixed sets, the CPE side could actually be running 15.2 and thus why it had the problem too. Either way, I know "line-rate" is the cause. My best guess is that 15.2 introduced a bug with it.